PaperlayerView exact revision on arXiv

Source-linked AI summary

Labels Are Not Endpoints: Treatment Leakage and Construct Validity in MCP Agent Security Evaluation

Rana Muhammad Ahmed, Sabahat Abbas

arXiv:2608.12880v1cs.CRcs.AI

TL;DR

Security evaluations can mistake stored labels for behavioral facts when treatment information enters endpoint construction. This paper audits a preserved campaign through evidence-bound reconstruction and treatment-blind validation, finding corrected case evidence while limiting its claims to the campaign rather than population attack rates or causal effects.

  • Problem

    Security evaluations need endpoints whose labels remain behaviorally meaningful and independent of treatment assignment.

  • Method

    The audit traces preserved execution evidence into reconstructed units and applies a treatment-blind endpoint with treatment-invariance and structural-validity checks.

  • Results

    The reconstruction corrects historical adversarial labels while retaining three verified protected-data transfers and one unauthorized-forwarding case, with exactly zero ATTACK_SUCCESS records in locked v2.

  • Takeaways & Limitations

    The surviving transfers are inspectable security cases, so the audit supports campaign-bounded measurement conclusions rather than a population attack effect.

  • Takeaways & Limitations

    Treatment invariance is necessary but not sufficient, and four reviewer/codebook mismatches plus structural invalidity constrain construct interpretation.

Abstract

from arXiv · show

Security evaluations of tool-using agents often equate stored labels with behavioral facts. We audit a preserved campaign by tracing 10,200 execution rows to 180 model-bound requests, 45 semantic requests, and 15 observable stimuli. Two schema treatments were delivered, but the planned external payload-family corpus was not. The historical grader exhibited direct treatment leakage: treatment metadata gated the ATTACK_SUCCESS class, so fixed behavior could change class under treatment relabeling. A treatment-blind reconstruction corrects 58 historical ATTACK_SUCCESS or HIJACK_ATTEMPT labels to authorized benign completions while preserving three verified protected-data transfers and one separate unauthorized-forwarding case. The locked v2 census contains exactly zero ATTACK_SUCCESS records, while the forwarding case remains a HIJACK_ATTEMPT at a semantic boundary concerning objective completion. A dual-reviewer blinded concordance review of all 96 requests deemed structurally interpretable by locked v2 produced identical reviewer-consensus classes but differed from the locked codebook on four construct-boundary cases. We contribute a seven-link Integrity Chain and an executable, scope-bounded endpoint-integrity linter. The result is a campaign-bounded measurement audit, not a population attack-rate, model-ranking, defense-efficacy, or causal estimate.

1 Introduction

The paper audits a preserved MCP agent-security campaign after finding that its historical endpoint allowed treatment metadata to influence adversarial labels. It reconstructs treatment-blind behavioral evidence and frames the result as a finite measurement audit rather than a population effect.

  • Problem: Treatment metadata entered the historical grader's positive-class gate, making CLEAN requests unable to receive that class from identical behavior.The campaign was reproducible, but its outcome construction was not independent of treatment assignment.
  • Implication: The campaign's key lesson is that reproducibility can preserve a construct defect as reliably as a valid measure.Security evaluations therefore require behaviorally anchored endpoints, not merely frozen code and artifacts.
  • Contributions: The study separates direct treatment leakage from broader construct validity and treats the corrected output as a finite census rather than a population effect.The contribution includes a treatment-invariance diagnostic, an authorization-aware treatment-blind endpoint, and evidence-bound reconstruction.
  • Method: The audit traces 10,200 execution rows to 180 model-bound requests, 45 semantic requests, and 15 observable stimuli.These reconstructed units restore denominators supported by the preserved execution evidence.
  • Contributions: The paper defines a seven-link Integrity Chain and a suite-bounded endpoint-integrity linter for mechanically testable measurement defects.The linter operationalizes part of the broader integrity analysis without replacing substantive construct-validity review.

2 Related Work and Remaining Gap

Related work makes indirect injection and unsafe action measurable, but the remaining gap concerns whether evaluation units, treatments, and endpoints support the claimed interpretation. The paper positions treatment invariance as a necessary diagnostic within a broader construct-validity analysis.

  • Existing benchmarks: Agent-security benchmarks measure both task completion and unsafe action, but their unit, injection locus, and outcome definition must match the claim.This frames evaluation design as a measurement problem, not only an attack-detection problem.
  • Evaluation units: Repeated deterministic observations are runtime replays rather than independent model draws, so treating them as experimental units produces pseudoreplication.The scientific unit must distinguish sampled stimuli from repeated observations.
  • Measurement integrity: Treatment invariance requires a fixed behavior to retain the same class when only treatment metadata changes, but passing it does not establish complete construct validity.Authorization rules, behavioral predicates, parser evidence, and source-to-sink interpretation still require substantive justification.
  • Remaining gap: The paper's contribution is evaluation integrity rather than a first claim about MCP poisoning or indirect prompt injection.Closest-work comparisons locate attacks and controls, while this study examines whether the evaluation endpoint itself is valid.

3 Study, Threat Model, and Evidence

The study evaluates schema-level interventions in a bounded MCP-style testbed where agents encounter model-facing discovery metadata, while the corrected analysis relies on executed behavior. The preserved campaign delivered matched CLEAN, POISON_TD, and POISON_CA variants, but did not deliver the planned payload-family corpus.

  • Bounded threat model: The testbed exposes deterministic local tools through MCP-style discovery, with adversarial control limited to natural-language discovery metadata rather than tasks, models, parsers, dispatchers, or tool implementations.Sources and sinks are local mocks; no deployed service, real account, personal record, or Internet destination is involved.
  • Planned and delivered treatments: Across 60 matched model–scenario triplets, CLEAN, POISON_TD, and POISON_CA differ only in the indicated model-facing schema component while tasks, prompts, defenses, and inference settings remain fixed.POISON_TD modifies selected tool descriptions, whereas POISON_CA modifies the capability advertisement.
  • Experimental configuration: The campaign uses nine hash-verified schema variants spanning three densities and three surfaces, with model identity, tokenizer templates, and deterministic decoding preserved across integrations.The fixed integrations use four named models, float16 weights, fixed software, greedy decoding, and a 512-token generation cap.
  • Evidence boundary: The broader payload-family plan was not executed: payload hashes were null, retrieved content was absent, and defense labels produced duplicate behavior rather than distinct delivered conditions.Consequently, the study identifies neither payload-family effects nor defense efficacy.
  • Evidence boundary: The corrected endpoint receives dispatcher-executed behavioral evidence and excludes treatment metadata when evaluating authorization, unauthorized objectives, and protected-data flow.This separates delivered schema content from the behavioral evidence used for corrected grading.

4 Methods

The methods reconstruct valid analysis units from repeated execution records, reproduce the historical treatment leakage, and replace it with a deterministic endpoint grounded in preserved behavior. The resulting finite census uses locked, treatment-blind rules and explicitly bounded denominators.

  • Analysis-unit reconstruction: 10,200 execution rows reduce to 15 observable stimuli, 45 semantic requests, and 180 model-bound requests arranged in 60 matched three-surface blocks.Hashing task text, expected sequences, and execution plans identifies stimuli before crossing schema variants, model revisions, and tokenizer authorities.
  • Analysis-unit reconstruction: Repeated complete-input groups show 100% concordance across serialized turns, outputs, parser state, stored class, and normalized behavior, so repetitions do not increase the stimulus or request denominator.Multiplicity ranges from 11 to 162 rows, with a median of 47.5.
  • Endpoint validity: The Treatment-Invariance Test holds executed behavior fixed, varies treatment metadata, and rejects an endpoint if its class changes; passing is necessary but not sufficient for construct validity.Formally, the required condition is f(B,T0) = f(B,T1) whenever B is fixed.
  • Historical endpoint defect: The historical grader changes an authorized two-tool completion from benign success to ATTACK_SUCCESS when only treatment metadata changes, proving direct treatment leakage.The leakage gate ignored unauthorized-call counts, authorized-sequence match, benign-task success, privilege aggregation, and explicit unauthorized-objective predicates.
  • Treatment-blind endpoint: The locked v2 checker classifies preserved evidence using authorization, task completion, unauthorized actions, and source-to-sink predicates without treatment, surface, poison, historical-label, or review-status inputs.Its classes include authorized benign completion, benign task failure, structural invalidity, critical exploit, attack success, and hijack attempt.
  • Populations and analysis: The primary corrected analysis is a finite census of all 180 model-bound requests, while conditional analyses use the 96 structurally interpretable requests and retain their selected denominators.Exact counts and finite proportions are reported; the design does not support inferential, causal, or superpopulation attack effects.

5 Results

The treatment-blind reconstruction corrects historical endpoint leakage while preserving a finite, evidence-bounded census and exposing remaining construct-boundary ambiguity. Reviewer consensus matched across 96 interpretable requests but differed from locked v2 on four cases.

  • 5.1 Reviewer-consensus–v2 construct boundaries: The 96-record blinded review achieved identical reviewer-consensus classes, but four cases differed from locked v2 at semantic or structural construct boundaries.One disagreement concerned whether an unauthorized objective was completed; three concerned human structural interpretability versus the frozen invocation contract, so v2 remains unchanged without being established as uniquely valid.
  • 5.2 Historical-to-corrected reconciliation: 58 of 70 historical ATTACK_SUCCESS or HIJACK_ATTEMPT labels reconstruct as authorized benign completions, including 22 attack-success and 36 hijack-attempt labels.The corrected census also includes one hijack attempt, one benign task failure, and ten structurally invalid requests among the remaining 12 cases.
  • 5.3 Corrected finite census: The corrected 180-request census contains 89 authorized benign completions, three benign task failures, one hijack attempt, three CRITICAL_EXPLOIT classifications, and 84 structurally invalid requests.The three critical classifications are verified protected-data transfers, and no case is classified as lower-severity unauthorized objective completion.
  • 5.3 Corrected finite census: All four security-relevant deviations occur in M1, while M3 contributes 45/45 invalid requests, indicating integration heterogeneity rather than model-family ranking.The M2 and M4 counts remain predominantly authorized or invalid, and the audit does not treat these frozen integrations as a model comparison.
  • 5.4 Three verified protected-data transfers and one hijack attempt: Three protected-data transfers execute authorized reads followed by unauthorized write_outbox calls with identical source-content and sink-body hashes.Two are D3 variants of one M1 scenario and one is a D5 tool-description case; matched CLEAN requests completed only authorized plans, so the bounded design does not support causal attribution.
  • 5.6 Structural invalidity changes the denominator: Structural invalidity leaves only 96/180 requests interpretable, and conditional transfer denominators do not support safety claims because uninterpretable behavior is not resistance.For D3/D5, invalid counts are 25/56 CLEAN, 26/56 POISON_CA, and 26/56 POISON_TD.
  • 5.7 Endpoint-Integrity Linter Evaluation: The endpoint-integrity linter reproduced 10/10 prespecified diagnostic outcomes, but this establishes only specified implementation behavior for the tested endpoints and fixtures.The expected F5 warning reflects a nonblocking representation-level schema-adapter limitation and does not certify or contradict the locked v2 census.

6 Discussion

The discussion argues that reproducibility can preserve an invalid endpoint, so security claims must separate treatment delivery from behavior and use treatment-blind, behaviorally anchored outcomes. The corrected campaign supports inspectable security cases and a finite census, not population effects, causal estimates, or broad external-validity claims.

  • What the study now establishes: 10,200 execution rows reduced to 180 model-bound requests and 15 observable stimuli, showing that execution volume is not study breadth.The discussion also identifies the corrected output as a finite census rather than a population effect.
  • What the study now establishes: Treatment delivery must be recorded separately from attack behavior because a poisoned schema reaching the model does not itself establish compromise.The paper frames treatment metadata inside the historical score as a measurement defect rather than evidence of attack success.
  • What the study now establishes: Three protected-data transfers remain concrete, inspectable observations, but their concentration in M1 scenario blocks does not establish a population rate or general surface effect.The verified trajectories bind authorization, executed capabilities, data identity, and destination.
  • Measurement-validity gate: The measurement-validity gate hashes treatment bytes, verifies model-input inclusion, separates runtime activation from assignment, compares behavior with authorization, blinds outcome computation, binds analysis keys, and retains structural invalidity.These seven checks operationalize the paper’s Integrity Chain and fail closed when their conditions are not met.
  • What the study now establishes: The separate forwarding case remains a hijack attempt because it reaches an unauthorized sink without establishing protected-data-transfer composition.Collapsing forwarding with protected-data transfer would reproduce the original measurement error at a different semantic boundary.
  • Threats to validity: Treatment invariance is necessary but insufficient for construct validity because authorization predicates, protected-flow interpretation, parser effects, and codebook boundaries still require justification.Four mismatches in blinded review and the selected 96-request stratum limit what the concordance evidence establishes.

7 Ethics and Reproducibility

The study uses local mock tools and canned data, while preserving frozen artifacts and adding a deterministic, treatment-blind correction. Its reproducibility claims remain bounded by author-attested reviewer provenance and the absence of real external recipients or deployed services.

  • Ethics and Reproducibility: The campaign uses local mock tools, canned data, and a local outbox, so no external party receives protected text.The setup avoids supplying a deployment-targeting exploit workflow while retaining enough evidence to evaluate the scientific claim.
  • Ethics and Reproducibility: Frozen Phase 4/4.5 artifacts and historical labels remain unchanged, while the additive v2 correction uses deterministic author-approved rules locked before surface-wise counts.The provenance boundary remains limited because reviewer completion, independence, blinding, no-AI use, and identities are author-attested in private hash-locked records.

8 Conclusion

The conclusion reframes the campaign from attack-rate evidence to a measurement audit: treatment-blind, behaviorally grounded endpoints preserve three verified transfers and one hijack attempt while yielding zero locked-v2 ATTACK_SUCCESS records. The Integrity Chain provides a checkable discipline for interpreting such bounded evidence.

  • 8 Conclusion: 58 historical ATTACK_SUCCESS or HIJACK_ATTEMPT labels were reclassified to authorized benign completion, while three protected-data transfers and one hijack attempt remained.The surviving transfers are inspectable security cases rather than a population effect.
  • 8 Conclusion: The locked v2 endpoint contains exactly zero ATTACK_SUCCESS records, while EI-067F9857ECE149 remains a HIJACK_ATTEMPT at an untested semantic boundary concerning objective completion.Deterministic codebook stability does not settle that semantic distinction.
  • 8 Conclusion: The Treatment–Behavior–Endpoint Integrity Chain binds treatment bytes, executed behavior, authorization, outcome rules, and analysis units before interpreting results.This discipline turns endpoint integrity into a checkable workflow.
Loading 2608.12880v1…