PaperlayerView exact revision on arXiv

Source-linked AI summary

Ranking Is Not Deciding: Benchmark Shortcuts and Operating-Point Collapse in Query-Time RAG Poisoning Defenses

Pushkal Kumar, Arvin Sahaym

arXiv:2610.33849v1cs.CRcs.CLcs.LG

TL;DR

Query-time RAG poisoning filters are often evaluated on poison sets that carry construction cues, while deployed-point false-positive rates are frequently unreported. The paper profiles poison construction and tests cue-removing edits across corpora and defenses; deleting a cue can leave ranking performance partly intact while breaking the deployed decision. Its findings are limited by the evaluation’s scope, and the authors claim no defense.

  • Problem

    Published query-time RAG poisoning defenses report near-perfect detection, but evaluations vary in poison construction and often omit false-positive rates at deployed thresholds.

    Source: §IX · p. 12

  • Method

    The paper profiles poison sets using target-question containment and poison-to-clean length overlap, then evaluates payload-preserving edits that remove construction cues.

    Source: §I · p. 1, C. What makes a rung an attack · p. 5

  • Results

    Deleting the target-question prefix from the attack authors’ passages moves RAGuard from AUROC 0.953 to 0.758, while word-shuffled length padding raises the learned scorer’s deployed miss rate from 0.000 to 0.986 with AUROC at 0.953.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors conclude that detection results should be reported with the poison set’s construction profile and the false-positive rate at the deployed threshold.

    Source: §IX · p. 12

    The measurement covers four corpora and omits a reranker, an NQ length rung, other poison rates and trust settings, and several attack conditions; the adaptive results use one attacker tuned to the authors’ scorer and transferred to others.

    Source: §VIII · p. 12

Abstract

from arXiv · show

Query-time filters that screen each retrieved document before it reaches the generator are the dominant published defense against retrieval-augmented generation poisoning, and they report near-perfect detection on domain benchmarks. We show that this performance is predicted by how the poison set was built rather than by what the filter reads: removing a single construction cue destroys the deployed decision before it destroys the ranking. Two label-free statistics of a poison set and its clean corpus profile the cue: containment, how often the target question appears verbatim in the poison, and length overlap. PoisonedRAG's released code prepends the target question to every inserted passage. Of the 21 published defense evaluations we survey, 8 verifiably run poison built this way and 5 substitute a length cue; separately, 12 report no false-positive rate at the operating point they deploy. Deleting only that prefix from the attack authors' own passages (0.93 still retrieved for their target) takes RAGuard, a published content filter we re-implement, from AUROC 0.953 to 0.758 and its miss rate from 0.191 to 0.763. Across four corpora, four attack families and ten per-document defenses on one shared candidate set, the profile predicts where detection saturates. Padding medical poison to clean length with word-shuffled filler that carries no content takes TrustRAG's set-level stage from a miss rate of 0.000 to 0.827, and one learned scorer of ours from 0.000 to 0.986 (0.321 at a matched 5% false-positive rate) while its AUROC holds at 0.953. The ranking survives; the decision does not. A defense-aware attacker defeats our own scorer, so we claim no defense. A detection result is evidence only when it is reported with its poison set's construction profile and the false-positive rate at the threshold actually deployed; we release the script that computes the profile.

I. INTRODUCTION

RAG poisoning exploits the external corpus to steer answers, while query-time filters decide which retrieved documents reach generation. The paper argues that benchmark construction cues can drive reported detection and operating-point collapse, motivating evaluation that accounts for how poison was built.

A. Poisoning attacks on RAG

  • RAG poisoning can steer outputs for chosen queries through inserted corpus text without changing model weights.Five passages per target question have sufficed for over 90% attack success, while later work uses one passage or trigger-hidden payloads.Source: §I · p. 1

I. INTRODUCTION

  • Two construction cues—verbatim target-question containment and poison-to-clean length separation—predict which benchmarks saturate before a defense runs.The paper reports that PoisonedRAG’s released code prepends the target question to inserted passages.Source: §I · p. 1
  • Deleting the question prefix from the attack authors’ passages moves RAGuard from AUROC 0.953 to 0.758.The passages report this as the leading control, with every miss rate tied to the false-positive rate at which it was measured.Source: §I · p. 2
  • On medical poison padded to clean length with word-shuffled filler, the learned scorer’s miss rate rises from 0.000 to 0.986 at its deployed threshold while AUROC remains 0.953.At a matched 5% false-positive rate, its miss rate is 0.321; TrustRAG’s set-level stage rises from 0.000 to 0.827.Source: §I · p. 2
  • The authors report that a defense-aware attacker passes 0.977 of poisoned documents past their learned scorer and claim no defense.They also state that the profile is validated on four corpora and that a single-cell prediction is wrong in 40 of 90 cases.Source: §I · p. 2

B. Defenses by stage and granularity

  • Defenses act at different pipeline stages, from offline and retrieval-time filters to pre-generation, generation-time, and post-generation systems.The paper distinguishes defenses by both when they act and whether they decide on documents or retrieved sets.Source: B. Defenses by stage and granularity · p. 2

C. Lineage of the decision rule

The paper situates its admission rule in prior work and treats detection as a cost-sensitive classification problem. Its survey finds construction cues and missing deployed-point false-positive rates across published evaluations.

C. Lineage of the decision rule

  • The paper describes its decision-rule ingredients as established prior work and calls the system a query-time admission gate.It uses that term because the gate observes the query, matching the coverage-bound setting it cites [57].Source: C. Lineage of the decision rule · p. 2

D. Evaluating a security classifier

  • The paper frames detector evaluation around benchmark bias, shortcut features, base-rate problems, and the possibility that robustness reflects attack construction.It also notes prior work on ranking metrics masking behavior at deployable false-positive rates –.Source: D. Evaluating a security classifier · p. 3

E. The gap

  • Of 21 published defense evaluations, 8 verifiably use poison containing the target question and 5 substitute a length or repetition cue.The survey also finds that 12 report no false-positive rate at the deployed operating point.Source: E. The gap · p. 3

A. System and trust boundary

  • A query-time admission gate scores each retrieved candidate and chooses PASS, ESCALATE, or BLOCK; only passed documents enter the generator context.The deployment label belongs to the query path, not an individual document field.Source: A. System and trust boundary · p. 3

B. Adversary and defender

The evaluation fixes a threat model in which an attacker can control inserted document content, then tests payload-preserving edits that remove construction cues. Experiments use four corpora, four attack families, and a shared candidate set for ten per-document defenses.

B. Adversary and defender

  • The threat model lets the adversary choose text and self-declared fields for documents in controlled sources, but not model weights.The attacker can query the system for a hard-label indication of whether a document reached context; reported attacks are black-box unless otherwise identified.Source: B. Adversary and defender · pp. 4–5

C. What makes a rung an attack

  • A control removes one profiled construction cue while holding payload and attack family fixed; it counts as a successful attack only if retrieval, payload integrity, and gate passage all hold.Retrieval is re-measured after each edit because length or lexical changes can move a document out of the retrieved set.Source: C. What makes a rung an attack · p. 5

A. Corpora, poison, and defenses

  • The evaluation uses four corpora, 100 target questions per corpus, and 40 evaluation queries per cell; retrieval returns k=10 candidates.The authors characterize the scale as small and fixed, with 400 candidates per cell and seed.Source: A. Corpora, poison, and defenses · p. 5
  • Four attack families are tested, with poison regenerated from published templates in 15 of 16 corpus-family cells.The exception is NQ under PoisonedRAG, where the attack authors’ own text is used and its question prefix is removed for the leading control.Source: A. Corpora, poison, and defenses · p. 5
  • Ten defenses score an identical candidate set and return one keep/remove decision per document; the paper counts eleven table rows because its learned scorer appears at three thresholds.Three of the four attacks specify no source model, and source identities are supplied by the authors.Source: A. Corpora, poison, and defenses · p. 5

B. Metrics and statistics

The paper reports several classification and harm metrics, then sets out a cost-sensitive admission rule. Its threshold interpretation depends on stated assumptions, including calibration at the prevalence where the gate runs.

B. Metrics and statistics

  • The evaluation reports false-negative and false-positive rates at deployed and matched-FPR rules, AUROC, calibration error, and harm per 1,000 queries.Context contamination rate is reported separately in the appendix, rather than combined into a composite index.Source: B. Metrics and statistics · p. 5
  • Comparisons use five seeds and paired cluster bootstraps over 40 question clusters, with each cluster carrying all five seeds.The authors caution that per-seed standard deviation is not a precision claim because candidate sets are seed-invariant.Source: B. Metrics and statistics · p. 5

C. Cost-sensitive admission

  • Calibration assumes P(y=1 | p̂=t)=t at the gate’s deployment prevalence, but the paper says this assumption fails when prevalence shifts from scorer training to deployment.Evidence and source identity are also assumed conditionally independent given the label, and the rule assumes per-document costs are deployment-stationary.Source: C. Cost-sensitive admission · p. 6
  • Under assumptions A1, A3, and A4, the pointwise harm-minimizing rule blocks when p ≥ 1/(1 + ρd).Here ρd is the deployment’s miss-to-false-block cost ratio; the passage states the rule with adjudication cost unset.Source: C. Cost-sensitive admission · p. 6

D. The scored system

The scored system combines query-containment and document-length cues with embedding, lexical, and set-context features. The poison profile measures verbatim target-question containment and overlap with the clean corpus’s word-length range before a defense runs.

D. The scored system

  • The scorer uses near-verbatim query containment and log document length alongside embedding, lexical, and set-context features; a feature mask supports ablation.Its MLP trunk uses [512, 256, 128] layers and is temperature scaled; a gradient-boosted tree serves as an ablation arm.Source: D. The scored system · p. 6

A. Two construction quantities

  • Containment is the fraction of poison passages containing a normalized target question as a contiguous substring; length overlap is the fraction within the clean corpus’s 5th–95th percentile word-count interval.The clean spread ℓ95/ℓ5 is printed alongside overlap because a corpus with no length variance can make low overlap a corpus property.Source: A. Two construction quantities · p. 6

V. CUE PROFILES AND THE CONTROL LADDER

  • The two label-free quantities profile poison and clean text before a defense runs; neither reads a defense score or requires held-out labels.They bound how much a detection result can establish, but are not themselves defenses.Source: A. Two construction quantities · pp. 6–7, §V · p. 6

A. Two construction quantities

  • Question-targeted released attacks show containment 1.00 by construction, while TrojanRAG’s fixed-width chunks have length overlap 0.00; length alone reaches AUROC 1.000 on medical but 0.098 on financial.Length overlap is coarse, so the paper reports the signed one-feature AUROC alongside it.Source: A. Two construction quantities · p. 6

B. The control ladder

Removing the question prefix from PoisonedRAG’s released passages sharply weakens RAGuard, while controls isolate containment and length cues and test them across a shared candidate set. Results show corpus-level saturation patterns, but the profile does not predict individual defense outcomes.

B. The control ladder

  • The control ladder removes the target-question prefix, matches poison length with clean filler, then tests placebo fillers and style mimicry while tracking retrieval survival.It compares FNR changes at the deployed point with changes at matched FPR = 5%; domain-regime scorers use disjoint skeleton halves, while the general-regime scorer trains on HotpotQA and MS MARCO poison.Source: B. The control ladder · p. 7

A. Prediction, and the published construction

  • On templated medical poison, 4 of 8 distinct score systems have FNR 0.000, while DRS has FNR 0.700 at FPR 0.035 and peaks at AUROC 0.882.Table V reports deployed miss rate and false-positive rate alongside miss rate at a matched 5% false-positive rate.Source: A. Prediction, and the published… · p. 7, Table V · p. 8
  • RAGuard falls from AUROC 0.953 to 0.758 and from deployed miss rate 0.191 to 0.763 when the prefix is deleted; 0.93 of the passages still retrieve for their target.The NQ control deletes the prefix from the attack authors’ own passages and otherwise holds the question, retriever, and candidate set fixed.Source: A. Prediction, and the published… · pp. 7–9
  • TrustRAG removes 0.996 of PoisonedRAG’s poison as inserted and 0.980 as archived; RAGDefender removes 0.884 versus 0.830, while the prefix also changes retrieval upstream.TrustRAG’s top five contains 4.79 poison passages per query with the prefix and 2.40 without it; on the literature’s construction it returns an empty context on 84 of 100 queries.Source: A. Prediction, and the published… · pp. 8–9, Table VI · p. 8
  • The profile predicts which corpora saturate, including the released NQ poison, but not individual defenses: its direction is wrong at 40 of 90 points and remains a hypothesis based on four corpora.The paper treats zero-miss results above FPR 0.5 as blocking rather than detection.Source: B. The control ladder · p. 7, Figure 2 · p. 8, A. Prediction, and the published… · p. 9

B. The control ladder

Removing one construction cue can leave another load-bearing, and padding can separate a scorer’s ranking from its deployed decision. The strongest tested attack stacks cue removal and mimicry, although retrieval survival and corpus structure constrain its outcome.

B. The control ladder

  • Removing the prefix raises the scorer’s deployed miss rate by 0.421 on financial, while length matching raises it by 0.983 on medical and 0.036 on legal, but only 0.014 on financial.The two one-feature detectors likewise fail under the control removing their own cue and remain unaffected by the other control.Source: B. The control ladder · p. 9
  • With word-shuffled filler, the scorer’s deployed miss rate reaches 0.986 while AUROC remains 0.953, and TrustRAG’s miss rate reaches 0.827; this is not content dilution.Perplexity and RAGuard behave differently under fluent versus shuffled filler, so the paper withdraws their padding-collapse interpretation and reports bounds.Source: B. The control ladder · p. 9
  • On NQ, the miss rate rises from 0.118 to 0.948 under the strongest tested attack, with survival 0.862; on medical it rises from 0.000 to 1.000 at survival 0.958.Legal is an exception: its miss rate is 0.497 at survival 0.757, as fixed-width chunking makes padded poison costly to retrieve.Source: B. The control ladder · p. 9
  • A query-based attacker tuned to the learned scorer passes 0.977 of its documents, while baseline miss rates span 0.069–1.000, with the low end bought by blocking almost every benign candidate.The attack is a transfer attack for the baselines, so those rates are a lower bound.Source: B. The control ladder · p. 9
  • A single control can understate the problem on some corpora and overstate it on others, so the authors conclude that one ablation is never evidence.Their verdict is that the load-bearing cue is corpus-specific.Source: B. The control ladder · p. 9

C. The dissociation, and two collapse modes

The section separates ranking degradation from failure at the deployed threshold, identifying calibration and signal collapse as distinct modes. Its controls show that the cue carrying detection varies by corpus.

C. The dissociation, and two collapse modes

  • The load-bearing cue is corpus-specific: length carries medical, the prefix carries financial, and NQ’s length rung is inert because overlap is 1.00.Figure 3 reports miss rates across four payload-preserving edits; retrieval survival appears beneath each rung.Source: Figure 3 · p. 10
  • On medical length matching, the two negative-log-likelihood detectors collapse with clean filler and recover with shuffled or pseudo-word fillers, while our scorer and TrustRAG collapse under all three.Figure 4 contrasts AUROC loss with added miss rate and plots deployed-posterior reliability; AUROC alone cannot distinguish calibration from signal collapse.Source: Table VII · p. 10, Figure 4 · p. 10
  • With shuffled filler, our scorer retains AUROC 0.953 but misses 0.321 at matched 5% FPR and 0.986 at its deployed threshold.The filler preserves length and vocabulary but shuffles word order, leaving no coherent content.Source: Figure 4 · p. 10, C. The dissociation, and two collapse… · p. 11
  • ECE 0.471 marks the headline calibration-collapse row, while signal-collapse AUROC reaches 0.503 on medical, 0.518 on NQ, and 0.511 on financial.The paper distinguishes calibration collapse, where the ranking remains usable but the deployed point fails, from signal collapse, where ordering approaches chance.Source: C. The dissociation, and two collapse… · p. 11

D. Mechanism and repair

Feature ablations and a clean-score floor show that repair depends on which cue remains. Decision rules can be inert on benchmark cells, motivating a reporting protocol that exposes construction, operating points, and score variation.

D. Mechanism and repair

  • On length-matched medical, deleting the length feature cuts deployed miss rate from 0.986 to 0.204 with clean filler and 0.988 to 0.026 with shuffled filler, while AUROC rises from 0.885 to 0.992.A clean-quantile floor moves the miss rate from 0.983 to 0.750 for +1.47 points of FPR; with both cues removed, it changes miss rate by only -0.029.Source: D. Mechanism and repair · p. 11

E. Decision agreement and inertness

  • In 12 of 26 corpus-by-attack cells, the harm-blind, cost-derived, and trust-adjusted rules take bit-identical actions on every candidate.On templated medical poison, 0.00% of candidates fall in the decision band; the cell has 397 distinct score values.Source: E. Decision agreement and inertness · p. 11
  • Where scores are unsaturated, the cost-derived point reduces harm per 1,000 queries from 4195 to 2585 on length-matched legal and from 1600 to 900 on prefix-removed medical.On the benchmark the literature reports against, all three rules take bit-identical actions, so operating-point claims on those cells cannot be falsified by an experiment run on them.Source: E. Decision agreement and inertness · p. 11

VII. A REPORTING PROTOCOL FOR QUERY-TIME FILTERS

  • The protocol calls for publishing the poison’s construction profile, testing multiple controls with placebo fillers, and reporting deployed FNR/FPR and ECE alongside AUROC.It also recommends distinct-score counts, matched-FPR or stated-harm comparisons, and retrieval survival with payload preservation for evasion claims.Source: §VII · p. 11

VIII. LIMITATIONS AND CONCURRENT WORK

The authors bound the profile’s scope and the evaluation’s limitations, then conclude that detection results need to be read alongside poison construction and deployed false-positive rates. They also describe the released artifacts, model use, and ethical safeguards.

VIII. LIMITATIONS AND CONCURRENT WORK

  • The profile is validated on four corpora and is wrong at 40 of 90 individual points; three corpora use regenerated poison, and the evaluation has one retriever and no reranker.Adaptive results are lower bounds from one attacker, without an LLM-guided or human arm or the strongest learned generator.Source: §VIII · p. 12

IX. CONCLUSION

  • The conclusion is that detection results should be reported with the poison set’s construction profile and the false-positive rate at the deployed threshold.Deleting one cue can leave ranking degraded but destroy the deployed operating point; deleting both removes the ranking too.Source: §IX · p. 12

OPEN SCIENCE

  • The release includes scripts and artifacts to reproduce the experiments, but withholds MS MARCO passages and embeddings under its research terms and does not release trained scorer weights.The build script regenerates scorer weights on CPU.Source: OPEN SCIENCE · p. 13

LLM USAGE CONSIDERATIONS

  • No language model is part of the scorer; models are used for selected evaluation components, and the authors inspect outputs used for editorial purposes.The paper says its claims do not depend on the optional generator-side experiment.Source: LLM USAGE CONSIDERATIONS · p. 13

ETHICAL CONSIDERATIONS

  • The evaluation uses public benchmark data, involves no human subjects or live systems, and bounds deployment claims to simulated settings and specified safeguards.The human-review path is simulated, and the released poison generators are bound to public benchmark questions.Source: ETHICAL CONSIDERATIONS · pp. 13–14

APPENDIX A CLOSED FORMS FOR THE TRUST STATE

Appendix A derives closed forms for the trust update rule under its own generative model and reports simulation-based budgets and steady-state behavior. These results are not guarantees or detection results.

APPENDIX A CLOSED FORMS FOR THE TRUST STATE

  • The appendix’s closed forms describe an update rule under its own generative model, not guarantees or detection results; source identity is an exact label indicator in every measured condition.The quoted numbers are produced by simulation using hand-set scorer profiles.Source: Appendix A · p. 17
  • Under symmetric forgetting, evidence mass approaches M and reaches 95% of M after ⌈ln 0.05/ln λ⌉ events, approximately 3/(1 −λ).The appendix derives this from an affine recursion with fixed point M.Source: Appendix A · p. 17
  • The closed forms give consecutive poison-event and benign-event counts for trust to cross a target κ⋆ under symmetric forgetting.The stated counts apply from starting states on the respective sides of the target condition.Source: Appendix A · p. 17
  • At deployed base rate π0 = 0.343 with κ⋆= 1, the simulated poison budget is 19 adjudicated events, versus 3 at the hypothetical π0 = 0.05.The budget depends on the base rate, so the paper states it with each figure.Source: Appendix A · p. 17
  • Under asymmetric forgetting, recovery takes longer than the symmetric-form count because poison evidence decays at the slower rate λa.The trajectories are monotone in the number of events.Source: Appendix A · p. 17
  • 40 of 90 corpus–attack–defense points have a profiled direction that is wrong at the individual-point level, although the profile’s claim is at corpus level.Fig. 5 plots the poison-only cue against miss-rate change when that cue is removed at the deployed operating point.Source: Figure 5 · p. 18
  • Table IX compares defenses on released NQ poison with and without the question prefix prepended by the attack’s code.The accompanying table text identifies deployed-threshold FNR/FPR, AUROC, matched-5% FPR miss rate, and retrieval survival as reported measures.Source: Table IX · p. 18, Appendix A · p. 18
  • At the deployed base rate, the simulated honest-source crossing point is 0.115 versus 0.013 at the hypothetical 0.05 base rate.The same asymmetric forgetting that lowers the attacker’s crossing point also lowers the honest source’s at the same adjudication error rate.Source: Appendix A · p. 18

APPENDIX B FULL E1 GRIDS

The appendix documents how construction controls, deployed thresholds, and decision rules affect the reported defense outcomes. It also specifies which comparisons are paired and which defenses cannot be scored on the shared candidate set.

Scope and controls

  • Table XI excludes systems that expose no per-document admission decision or no sweepable operating point from shared-candidate-set scoring.Table X shows that legal padding overshoots the clean length band and installs an inverted cue rather than removing one.Source: Table XI · p. 19, Table X · p. 19

Paired comparisons

  • Table XII reports paired cluster-bootstrap differences against the harm-blind τ=0.5 classifier for legal/PoisonedRAG_BB.Intervals use 1,000 paired resamples of 40 query clusters pooled over five seeds; the table caption identifies the comparison scope.Source: Table XII · p. 19, Appendix B · p. 19

Decision agreement

Cost-derived operating points

  • On length-matched legal, wrapping the cost-derived rule over the chunk score and perplexity reduced harm per 1,000 queries from 18,460 to 3,915 and from 50,100 to 3,725, respectively.Both wrappers incur false-positive costs; unlike τd, the wrapper requires a labelled calibration set.Source: Appendix B · p. 20
  • On financial out-of-distribution data, the rule lowers harm by 26.6% but raises false-positive rate by 2.6 points; prior correction leaves harm unchanged at 730.The cost-derived threshold also has proportionality regret of 170.0 on financial and 20.0 on legal, versus 0.0 on medical.Source: Appendix B · p. 20

Adaptive attack comparison

  • Table XIV evaluates mimicry plus hard-label word substitution tuned against the scorer on NQ, with static/adaptive outcomes at deployed thresholds and transfer attacks for all but the last four rows.The attack uses archive-stored, prefix-less passages, 500 oracle queries, and 60 iterations per document.Source: Table XIV · p. 21

APPENDIX C PRIOR REVIEWS AND RESPONSE

The appendix records that an earlier version was rejected after two reviews and that its reported results were withdrawn following a code audit. The current submission reports results from a rewritten pipeline and identifies planned experiments cut for compute as limitations.

Review history

  • An earlier version was submitted to HICSS-60 and rejected after two reviews, which are reproduced in full.There was no rebuttal and no additional review.Source: Appendix C · p. 22

Audit and withdrawn results

  • A code audit confirmed reviewer concerns and found further defects, including an untrained ranking head and a baseline fit on evaluation labels.The audit also found a reliability signal derived from filter decisions and hard-coded strings from the authors’ attack generator.Source: Appendix C · p. 22

Revised submission

  • All earlier-version numerical results are withdrawn; this submission reports only the rewritten pipeline’s results on public corpora with published attacks.Experiments E5 and E6 were cut for compute and named as limitations.Source: Appendix C · p. 22

A. How each criticism is addressed

The response describes a substantial rewrite after reviewers challenged the methodology, baselines, and scope of the claims. It records retired components, revised comparisons, and experiments or settings retained as limitations.

Methodology and decision rule

  • The revised work frames the method as a cost-sensitive decision problem with explicit assumptions, and measures the effect of the harm ratio with the scorer fixed.The pass rule is replaced by τd,s = 1/(1 + ρdκs); the untrained heads and self-derived reliability signal are removed, and trust updates use an exogenous adjudication ledger.Source: A. How each criticism is addressed · p. 22

Baselines and metrics

  • Baselines now share one filter interface and candidate set, with detector metrics limited to systems that emit labels.Same-task baselines include perplexity, RAGuard, DRS, TrustRAG-S1, RAGDefender, NLI consistency, and a uniform-threshold classifier; systems requiring a GPU or generator internals are capability-only comparisons.Source: A. How each criticism is addressed · p. 22

Contribution and related work

  • The paper no longer claims a system-level contribution; its stated contributions are the construction profile, two failure modes and their protocol, and measurement of the decision layer.The cost-sensitive threshold, Beta trust, per-document decision, and audit record are treated as instrumentation credited to prior work.Source: A. How each criticism is addressed · p. 22

Trust and provenance limits

  • Trust updates are non-automatic, and trust-related pipeline results are reported at ε=0; the legitimate-update benchmark was cut for compute.The paper states that content-only signals cannot distinguish identical texts.Source: A. How each criticism is addressed · p. 22

Claim scope

  • The response narrows the claims to partial support in the medical condition and rejects superiority or deployment-readiness claims.Adaptive-attack success is reported even when high.Source: A. How each criticism is addressed · p. 23

Threat-model boundaries

  • Index and retriever tampering are out of scope, and vector-database manipulation is removed from the defended classes.The response also states that no claim is made on an unfalsifiable rung and scopes profile predictions to the corpus level with per-point failure counts.Source: A. How each criticism is addressed · p. 23

Evaluation scope

  • The poison-rate sweep was cut for compute, so one poison rate is used throughout; injection budgets are instead compared across attack families at N=1 and N=5.The response says primary metrics are prevalence-invariant and use an odds correction.Source: A. How each criticism is addressed · p. 23
  • The response retains four real corpora and published attacks, but identifies a single retriever and 40 evaluation queries per cell as limitations.The main grid uses five seeds, the ablation three, and the secondary E9 one, as stated where they appear.Source: A. How each criticism is addressed · p. 23

B. The reviews

The section reproduces two anonymized reviews, with typesetting changes and disclosed name redactions. Both reviewers find the framework promising but say technical ambiguities and limited evaluation substantially narrow its supported claims.

B. The reviews

  • The reviews are presented as complete and otherwise unedited, with the system name replaced for anonymity and mechanical typesetting changes disclosed.The omitted cover email contained no review content.Source: B. The reviews · p. 23
  • Reviewers question the decision logic, including a pass rule that may loosen filtering as risk rises and an unclear combined score and training objective.They also seek clarification of how source trust is updated and how legitimate document revisions are distinguished from poisoning.Source: B. The reviews · pp. 23, 24, 26
  • Review 2 says the related-work section does not compare relevant runtime defenses, leaving novelty and baseline choices insufficiently established.Suggested comparators include document-level filtering, source-reliability systems, embedding-space defenses, and attribution methods.Source: B. The reviews · pp. 24, 26
  • The reviews request a clearer architecture, filtering equations, score semantics, training objectives, and operational definition of the reliability signal.They warn that erroneous trust updates could reinforce later filtering decisions and ask how authorized corrections and superseded documents are handled.Source: B. The reviews · pp. 24, 26
  • The baseline comparison conflates document-labeling capability with security outcomes, making zero F1 values misleading for systems that emit no document labels.The reviews recommend comparing shared outcomes such as poison pass rate and marking F1 not applicable where labels are unavailable.Source: B. The reviews · pp. 23, 25, 26
  • The reviewers say domain results provide only partial evidence of proportionality: medical performance improves, while financial and legal gains are negligible or unsupported.They also question the derivation of domain-risk weights and the definitions or interpretation of DPS, TCA, GSQ, and detection rate.Source: B. The reviews · pp. 25–26
  • The evaluation omits vector-database manipulation and does not establish that findings generalize beyond its synthetic corpus, limited conditions, and small number of trials.The reviewers also request sensitivity analysis for the unintended change in poison prevalence.Source: B. The reviews · pp. 25–26
  • Reviewers recommend treating the proposed thresholds, review cadences, and escalation values as illustrative because the study does not derive or validate them.They also recommend revising or removing figures that duplicate tables or display unavailable F1 values as zero performance.Source: B. The reviews · pp. 25–26
Loading 2610.33849v1…