PaperlayerView exact revision on arXiv

Source-linked AI summary

Poisoning Retrieval Corpora by Injecting Adversarial Passages

Zexuan Zhong, Ziqing Huang, Alexander Wettig, Danqi Chen

arXiv:2310.19156v1cs.CLcs.IR

TL;DR

Dense retrievers are powerful but their safe real-world deployment remains uncertain. The paper proposes corpus poisoning through adversarial passages optimized over discrete tokens, finding that small injections can mislead dense retrievers on unseen and out-of-domain queries while exposing practical computational and scope limitations.

  • Problem

    Dense retrievers achieve strong information-retrieval performance, but it remains unclear how safely they can be deployed in real-world applications.

  • Method

    The attack generates adversarial passages by perturbing discrete tokens to maximize similarity to training queries, then injects them into the retrieval corpus.

  • Results

    Small numbers of injected passages successfully attack state-of-the-art dense retrievers and generalize to unseen queries from out-of-domain corpora.

  • Takeaways & Limitations

    The findings have implications for deploying robust retrieval systems in real-world applications and for improving their safety and robustness.

  • Takeaways & Limitations

    The approach can be computationally expensive, and the study specifically investigates corpus poisoning rather than other attack avenues such as malicious query manipulation.

Abstract

from arXiv · show

Dense retrievers have achieved state-of-the-art performance in various information retrieval tasks, but to what extent can they be safely deployed in real-world applications? In this work, we propose a novel attack for dense retrieval systems in which a malicious user generates a small number of adversarial passages by perturbing discrete tokens to maximize similarity with a provided set of training queries. When these adversarial passages are inserted into a large retrieval corpus, we show that this attack is highly effective in fooling these systems to retrieve them for queries that were not seen by the attacker. More surprisingly, these adversarial passages can directly generalize to out-of-domain queries and corpora with a high success attack rate -- for instance, we find that 50 generated passages optimized on Natural Questions can mislead >94% of questions posed in financial documents or online forums. We also benchmark and compare a range of state-of-the-art dense retrievers, both unsupervised and supervised. Although different systems exhibit varying levels of vulnerability, we show they can all be successfully attacked by injecting up to 500 passages, a small fraction compared to a retrieval corpus of millions of passages.

1 Introduction

The paper identifies a corpus-poisoning vulnerability in dense retrievers: malicious users can inject a small number of adversarial passages optimized for broad query sets, including unseen and out-of-domain queries. Across retriever types, the attack can substantially alter retrieval outputs, raising safety concerns for real-world deployment.

  • Dense retrievers outperform traditional lexical methods across information-retrieval tasks, but their safe deployment in real-world applications remains unclear.
  • The attack injects a small fraction of adversarial passages into a retrieval corpus to make dense retrievers return them among top results.
  • Adversarial passages are optimized by iteratively perturbing discrete tokens to maximize similarity to a set of training queries, with clustering used to generate multiple passages.
  • Even 10 passages fool more than 90% of queries for unsupervised Contriever, while supervised retrievers exceed 50% success when up to 500 passages are injected, representing <0.02% of the corpus.
  • The generated passages generalize to unseen in-domain and out-of-domain queries, including queries over financial documents or online forums.
  • A case study examines targeted misinformation as a real-world threat from inserted text and motivates greater attention to dense-retrieval safety and robustness.

2 Related Work

The paper distinguishes corpus poisoning from earlier adversarial retrieval, data poisoning, image retrieval attacks, and universal triggers. Its defining setting is injecting new passages into an unchanged retrieval corpus and evaluating system-level success on unseen queries.

  • Adversarial attacks in NLP classification and question answering generally seek small semantic-preserving perturbations that induce erroneous predictions.
  • Earlier retrieval attacks edit passages to change rankings for individual or small query sets, whereas corpus poisoning injects new passages and evaluates overall performance on unseen queries.
  • Corpus poisoning changes only the retrieval corpus, not training data, and leaves the retriever unchanged, distinguishing it from data poisoning.
  • Text attacks require optimization over discrete tokens, unlike image-retrieval attacks that optimize continuous pixel values.
  • Universal adversarial triggers alter every input sequence, while retrieval poisoning changes system outputs by injecting new passages without modifying passages or user queries.

3 Method

The method constructs a small adversarial passage set by maximizing similarity to training queries and inserting the resulting passages into the corpus. Gradient-based discrete-token optimization is extended through query clustering to target multiple groups in parallel.

  • For each query q, dense retrievers encode q and passages separately and rank passages using the inner product Eq(q)^⊺Ep(p).
  • The attack generates a small set A of adversarial passages, with |A| ≪ |C|, and inserts them into corpus C so at least one reaches the top-k results for queries in Q.
  • The optimization searches for a token sequence that maximizes similarity to a set of queries, using held-out queries to test generalization.
  • Starting from a random corpus passage, the gradient-based HotFlip procedure repeatedly selects a token and evaluates replacement tokens through an approximation of model-output change.
  • The replacement search uses gradients with respect to the selected token embedding to approximate which vocabulary token will maximize similarity after replacement.
  • To create multiple passages, k-means clusters queries by Eq(qi) embeddings and generates one adversarial passage per cluster in parallel.

4 Experiments

The experiments evaluate corpus poisoning across retrieval datasets, dense-retriever architectures, passage counts, passage lengths, and targeted misinformation. The attack transfers across domains, affects multi-vector retrievers, and remains effective under constrained settings.

  • Experimental setup: The evaluation covers NQ and MS MARCO held-out queries plus transfer to seven unseen BEIR domains, including financial documents.Attacks target five dense retrievers and additionally evaluate ColBERT.
  • In-domain attacks: More than 75% of NQ and MS MARCO queries are fooled by pre-trained Contriever with only one adversarial passage.Supervised retrievers are harder to attack, but generating more passages substantially improves success.
  • Out-of-domain transfer: A 94.1% attack success rate is reported on FiQA when adversarial passages generated from NQ or MS MARCO transfer to other BEIR domains.The transfer experiment uses Contriever and 50 adversarial passages.
  • Multi-vector retrieval: ColBERT is fooled on 20.1% of NQ test queries by inserting 50 adversarial passages of 250 tokens.Longer passages are more effective for ColBERT, consistent with its multi-vector similarity computation.
  • Targeted misinformation: With a fixed targeted-misinformation prefix, success rates reach 11.7% with one passage and 59.6% with ten passages.The targeted message is designed to appear after two or three queries.

5 Additional Analysis and Discussion

Additional analyses show that adversarial passages have limited cross-model transfer, can remain effective with sufficient length or nonsensical initialization, and do not transfer to BM25. Transfer across domains is not explained simply by train-test query overlap.

  • Attacks are not model-agnostic: Transfer success rates remain below 0.5% across similar models and show no transferability across different model families.This limits the usefulness of transferability for black-box attacks against systems with unknown model weights.
  • Lengths of adversarial passages: Adversarial passage length has little effect on attack performance once passages contain at least 20 tokens.The analysis attacks Contriever-ms with 50 passages of lengths 10, 20, 50, and 100 using Natural Questions training data.
  • Initialization of adversarial passages: 95.8% attack success remains after initializing passages with 50 [MASK] tokens, compared with 98.1% using natural passages.The result indicates that nonsense initialization remains effective, although natural passages improve the attack.
  • Unnaturalness of adversarial passages: The attack produces unnatural passages that enable simple defenses, although stronger attacks may generate fluent text that still fools models.The authors identify unnaturalness as a current weakness while noting a potential path for stronger attacks.
  • Transferability does not simply come from train-test overlap: Adversarial passages transfer across domains and datasets, and this transferability does not simply result from overlap between training and test queries.The paper reports these findings across its main analysis and additional transfer attack strategies.

6 Conclusions

The paper introduces corpus poisoning against dense retrievers and shows that small numbers of injected adversarial passages can attack state-of-the-art systems and generalize to unseen domains. These findings bear on the deployment of robust retrieval systems in real-world applications.

  • 6 Conclusions: The proposed attack inserts adversarial passages into dense-retrieval corpora to mislead retrieval outputs.The attack targets the corpus rather than changing the retriever itself.
  • 6 Conclusions: A small number of adversarial passages can successfully attack state-of-the-art dense retrievers and generalize to queries from unseen domains.The conclusion frames these results as relevant to future deployment of robust retrieval systems in real-world applications.

Limitations

The study identifies computational cost and scope as key limitations: generating passages requires independent GPU-based attacks, while the investigation focuses specifically on corpus poisoning rather than other attack avenues.

  • Limitations: Generating one adversarial passage for each query group requires running the attack independently on a single GPU, making the approach computationally expensive.The authors suggest developing more efficient methods for generating multiple adversarial passages.
  • Limitations: The study focuses on corpus poisoning attacks and does not investigate other potential avenues, such as malicious manipulation of user queries.The stated scope concerns the implications of corpus poisoning for dense-retriever vulnerability.

Ethical Considerations

The ethical discussion emphasizes that the attack can mislead widely used dense retrieval systems and may be misused to spread toxic information. The authors therefore call for caution in future work and disclose a small misinformation example used to illustrate the threat.

  • Ethical Considerations: Malicious users can successfully perform the proposed attack, producing a high rate of misleading dense-retrieval outputs.The authors frame this risk in the context of dense retrieval models widely used in industry.
  • Ethical Considerations: The attack could be misused to spread toxic information, so future research should consider the potential consequences of proposed methods.The paper also describes including a small misinformation example to highlight corpus poisoning's harmful potential.

A Dataset Details

The evaluation uses BEIR retrieval datasets spanning multiple tasks and domains, with experiments centered on Natural Questions and MS MARCO and transfer tests across seven unseen domains.

  • A Dataset Details: The study uses BEIR datasets covering diverse retrieval tasks and domains.The listed datasets include Natural Questions, MS MARCO, HotpotQA, FiQA, Quora, FEVER, TREC-COVID, ArguAna, and SCIDOCS.

B Implementation Details

The attack optimizes fixed-length adversarial passages through iterative gradient-guided token replacement, selecting substitutions that increase similarity to query batches.

  • B Implementation Details: Adversarial passages are fixed at length 50 and optimized for 5,000 token-replacement steps.Each optimization step updates one randomly selected token.
  • B Implementation Details: Each step computes gradients over a query batch of size 64.The batch determines the similarity objective used to guide token replacement.
  • B Implementation Details: Gradient-based HotFlip generates replacement candidates for the selected token.The candidate set is chosen to maximize approximate post-replacement similarity.
  • B Implementation Details: The attack considers the top 100 candidate tokens and applies the candidate producing the greatest measured similarity increase.Actual similarity changes are evaluated before committing the replacement.

C Defenses

The paper examines defenses based on detecting unnatural language and unusually large embedding norms, while also probing transfer behavior and the qualitative structure of generated passages. Norm clipping substantially reduces attack success with limited retrieval degradation, but transfer depends on domain similarity and some architectures may remain harder to detect.

  • C Defenses: Adversarial passages contain unnatural sequences and unusually high embedding ℓ2-norms.These properties motivate likelihood filtering and embedding-norm clipping as defenses.
  • C Defenses: Figure 3 compares average token log likelihood for Wikipedia passages with 10 corresponding adversarial passages.An off-the-shelf GPT-2 can separate the two passage groups almost perfectly, although natural-language priors could potentially weaken this defense.
  • C Defenses: Passage embedding norms can inflate inner-product similarity, motivating clipping at a constant α across the retrieval corpus.The paper links this defense to the relationship sim(p,q) ∝ ∥E_p(p)∥2 cosθ.
  • C Defenses: α = 1.75 reduces attack success rate by 99.4% while hurting standard retrieval performance by 6%.The result is reported for Contriever on NQ using the clipping experiments summarized in Table 5.
  • C Defenses: SCIDOCS attack performance drops from 76.1% to 25.2% when transfer uses only the most similar training query or k-means cluster.HotpotQA declines less because both it and NQ contain knowledge-related Wikipedia questions.
  • C Defenses: Generated passages are difficult to interpret as human text but contain query-relevant keywords, while ColBERT may be harder to detect through norm anomalies.The examples include topic-linked terms such as “bulls” in basketball-related groups.
Loading 2310.19156v1…