PaperlayerView exact revision on arXiv

Source-linked AI summary

Applying Pre-trained Multilingual BERT in Embeddings for Improved Malicious Prompt Injection Attacks Detection

Md Abdur Rahman, Hossain Shahriar, Fan Wu, Alfredo Cuzzocrea

arXiv:2409.13331v1cs.CLcs.CR

TL;DR

Prompt injections can hide malicious instructions in LLM inputs, creating a detection challenge for applications. This paper tokenizes and embeds prompts with multilingual BERT, then evaluates machine-learning classifiers. Logistic Regression with multilingual BERT embeddings achieves 96.55% accuracy in the reported results.

  • Problem

    Prompt injections can make malicious inputs difficult to distinguish from legitimate commands, while attack goals and evaluation approaches vary.

    Source: §I · p. 1

  • Method

    The study tokenizes and embeds prompts with multilingual BERT and uses the resulting representations to train and evaluate machine-learning classifiers for malicious-versus-legitimate prompt classification.

    Source: §VI · p. 4

  • Results

    96.55% accuracy was achieved by Logistic Regression, which also had precision 1.00, recall 0.9333, and F1-score 0.9655.

    Source: B. Results · p. 6, §VIII · p. 6

  • Takeaways & Limitations

    The authors conclude that multilingual BERT embeddings notably enhance Logistic Regression performance for detecting malicious prompts.

    Source: §VIII · p. 6

Abstract

from arXiv · show

Large language models (LLMs) are renowned for their exceptional capabilities, and applying to a wide range of applications. However, this widespread use brings significant vulnerabilities. Also, it is well observed that there are huge gap which lies in the need for effective detection and mitigation strategies against malicious prompt injection attacks in large language models, as current approaches may not adequately address the complexity and evolving nature of these vulnerabilities in real-world applications. Therefore, this work focuses the impact of malicious prompt injection attacks which is one of most dangerous vulnerability on real LLMs applications. It examines to apply various BERT (Bidirectional Encoder Representations from Transformers) like multilingual BERT, DistilBert for classifying malicious prompts from legitimate prompts. Also, we observed how tokenizing the prompt texts and generating embeddings using multilingual BERT contributes to improve the performance of various machine learning methods: Gaussian Naive Bayes, Random Forest, Support Vector Machine, and Logistic Regression. The performance of each model is rigorously analyzed with various parameters to improve the binary classification to discover malicious prompts. Multilingual BERT approach to embed the prompts significantly improved and outperformed the existing works and achieves an outstanding accuracy of 96.55% by Logistic regression. Additionally, we investigated the incorrect predictions of the model to gain insights into its limitations. The findings can guide researchers in tuning various BERT for finding the most suitable model for diverse LLMs vulnerabilities.

I. INTRODUCTION

Prompt injections threaten LLM applications by hiding instructions in external data, making malicious inputs hard to distinguish from legitimate commands. The introduction describes varied attack goals and limits of handcrafted attack prompts as motivation for detection research.

I. INTRODUCTION

  • Prompt injections can hide instructions in external data, making malicious inputs difficult to distinguish from legitimate user commands and threatening LLM application behavior.The introduction also notes OWASP listed prompt injection among the top ten security concerns.Source: §I · p. 1
  • Attack goals include goal hijacking, which seeks an unwanted specific outcome, and prompt leaking, which seeks disclosure of previous messages such as system prompts.The introduction says differing goals and evaluation approaches make the purpose of attacks difficult to characterize.Source: §I · p. 1
  • Prior prompt-injection studies identify diverse goals, complicating the creation of a unified and comprehensive evaluation process.The introduction also situates the topic alongside earlier work on network security and machine-learning-based cyberattack detection.Source: §I · p. 1
  • The paper motivates multilingual BERT embeddings as an approach to detecting harmful prompt injections across languages and settings.It connects improved detection with safer LLM use in areas including healthcare, finance, and customer service.Source: §I · p. 2
  • Reported handcrafted techniques include special characters, context-switching text, and role-playing or instruction-updating scenarios.The introduction cites Yi et al. (2023) for special characters and Toyer et al. (2023) for an online game collecting handcrafted prompts.Source: §I · p. 2
  • Handcrafted prompts restrict attack range, vary in effectiveness across instructions and data, and complicate adaptive attacks, potentially overstating defenses.The paper notes these limitations can make assessments difficult.Source: §I · p. 2

II. PROMPT INJECTIONS

The paper describes prompt injections as malicious prompts that can redirect LLM outputs or alter LLM-integrated programs through Internet content. It also introduces NLP as computational processing of human language.

II. PROMPT INJECTIONS

  • Prompt injections can manipulate LLM outputs through crafted prompts that redirect an application from its intended response.The example prompt instructs an application to print “hello world” instead of its intended response.Source: §II · p. 2
  • Attackers can also embed malicious payloads in Internet resources that LLM-integrated programs process, altering application behavior.The passage connects this exposure to applications communicating with the Internet.Source: §II · p. 2

III. NATURAL LANGUAGE PROCESSING

  • Natural language processing (NLP) enables computational systems to communicate effectively with human language.The paper names chatbots, translation, and sentiment analysis as application areas.Source: §III · p. 2
  • NLP systems parse text and speech to derive meaning and transform raw language into actionable insights.The passage also describes uses in knowledge extraction and automated communication.Source: §III · p. 2

A. Tokenization

This section introduces tokenization and term-frequency weighting as ways to represent text for language processing. It explains TF-IDF as the product of term frequency and inverse document frequency.

A. Tokenization

  • Tokenization breaks text into smaller units, from characters to words, to make language manageable for machine analysis.The paper illustrates word-level and character-level tokenization and notes that granularity can suit particular languages and tasks.Source: A. Tokenization · p. 2

B. Term Frequency

  • Term frequency (TF) measures how often a term appears in a document, calculated as its count divided by the document's total number of terms.The notation identifies n_t,d as the term count and N_d as the total number of terms.Source: B. Term Frequency · p. 2
  • Figure 1 depicts steps for embedding prompt texts into vectors using multilingual BERT.The figure is referenced in the term-frequency subsection.Source: B. Term Frequency · p. 3
  • Inverse document frequency (IDF) reduces the weight of ubiquitous terms and increases the weight of rare terms in a corpus.The supplied formula introduction identifies IDF as a term-level quantity.Source: B. Term Frequency · p. 3
  • TF-IDF is obtained by multiplying TF by IDF.The section also introduces a sample-corpus calculation with N=4 documents.Source: B. Term Frequency · p. 3

C. Word Embedding

The section presents word embeddings as vector representations of words that capture semantic relationships. It contrasts them with TF-IDF on semantic similarity, out-of-vocabulary handling, and pretrained representations.

C. Word Embedding

  • Word embeddings represent words as vectors in a continuous space, with dimensions encoding word-vector values.The section describes these vectors as numerical representations of semantic similarities and relationships based on occurrence patterns.Source: C. Word Embedding · p. 3
  • Words with similar contexts can have closely aligned embedding vectors, capturing semantic meaning and relationships.The passage gives “car” and “vehicle” as words that embeddings can place close together.Source: C. Word Embedding · p. 3
  • The section says embeddings can handle out-of-vocabulary words, use pretraining, and capture relations that TF-IDF treats separately or does not represent.It presents these properties as advantages for text classification.Source: C. Word Embedding · p. 3

IV. BERT

BERT uses surrounding text in both directions to represent language context. Its pretraining and fine-tuning support contextual understanding across text-classification tasks.

  • BERT is presented as a transformer-based, pretrained framework for understanding language through surrounding text.Source: §IV · pp. 3–4
  • BERT processes text bidirectionally, considering context from both directions to capture language nuances and dependencies.Source: §IV · p. 3
  • Masked Language Modeling trains BERT to predict masked words from surrounding context, while Next Sentence Prediction distinguishes sequentially related sentences from unrelated ones.Source: §IV · pp. 3–4
  • BERT can be fine-tuned on specific datasets, such as question-and-answer pairs, for applications including text classification and machine translation.Source: §IV · p. 4

V. DATASET

The study uses the deepset Prompt Injection Dataset, with prompts and labels, to train and test classifiers. Multilingual BERT tokenizes and embeds prompts, and those vectors become input features for machine-learning algorithms.

  • The dataset contains 662 samples, split into 546 training and 116 testing examples, and the original holdout division is retained.Source: §V · p. 4
  • Prompts are mostly in English, with translations in other languages such as German, and the dataset has text and classification-label columns.Source: §V · p. 4
  • Multilingual BERT tokenizes prompt text into subwords and produces semantic embeddings that serve as features for machine-learning classifiers.Source: §V · p. 4

VI. METHODS

The method applies pretrained multilingual BERT to prompt-injection classification across languages. It converts prompts into contextualized vectors for machine-learning models, then evaluates their detection performance on testing data.

  • The approach uses multilingual prompt samples and pretrained multilingual BERT for prompt classification.Source: §VI · p. 4
  • Multilingual BERT embeddings are selected because the dataset includes several languages, unlike the language-specific Word2Vec embeddings discussed.Source: §VI · p. 4
  • The pipeline tokenizes prompts into contextualized vector representations, feeds them to machine-learning models, and evaluates detection using testing data.Source: §VI · p. 4

VII. RESULTS AND DISCUSSION

The section describes accuracy, precision, recall, and F1-score as measures for assessing prompt-injection detection. It defines accuracy through correct and incorrect classifications and explains the roles of precision and recall in interpreting F1.

A. Accuracy Metrics

  • Accuracy is calculated from true positives and true negatives relative to all true and false positive and negative classifications.Source: A. Accuracy Metrics · p. 4
  • Figure 3 presents ROC and AUC curves for Gaussian Naive Bayes, Random Forest, and Logistic Regression across its panels.Source: Figure 3 · p. 5
  • Precision describes the accuracy of favorable predictions, while recall measures the share of actual positive cases correctly identified.Source: A. Accuracy Metrics · p. 5
  • F1-score ranges from 0 to 1, with precision and recall helping explain the score.The section says these statistics assess how well the system identifies malicious prompt attacks.Source: A. Accuracy Metrics · p. 5
  • Figure 4 is a bar diagram of accuracy metrics for various machine-learning models.Source: Figure 4 · p. 5

B. Results

The proposed multilingual-BERT approach with Logistic Regression outperformed the prior methods on the prompt-injection dataset. Among the evaluated classifiers, Logistic Regression had the strongest reported metrics, while multilingual BERT was judged better suited than DistilBERT for multilingual prompts.

B. Results

  • For multilingual prompt detection, the authors judged bert-base-multilingual-uncased better suited than DistilBERT, which they describe as primarily trained on English.DistilBERT has fewer parameters and is optimized for efficiency and speed.Source: B. Results · p. 6
  • Logistic Regression achieved accuracy 0.9655, precision 1.00, recall 0.9333, and F1-score 0.9655 among the evaluated models.The reported model metrics are presented in TABLE I.Source: B. Results · p. 6, Table I · p. 6
  • Naive Bayes achieved accuracy 0.8879, while Random Forest achieved accuracy 0.8965 and precision 1.00.Random Forest recall was 0.80 and F1-score was 0.8888.Source: B. Results · p. 6
  • 96.55% accuracy: multilingual BERT with Logistic Regression significantly outperformed previous methods on the prompt-injection dataset.The comparison covers models and accuracy across different datasets.Source: Figure 5 · p. 6, B. Results · p. 6, Table II · p. 6

VIII. CONCLUSION

The study evaluates BERT-based methods for classifying malicious versus legitimate prompts and reports benefits from multilingual BERT embeddings. Logistic Regression with those embeddings achieved 96.55% accuracy.

VIII. CONCLUSION

  • The study applied multilingual BERT and DistilBERT to classify malicious versus legitimate prompts and examined tokenization and multilingual-BERT embeddings.Source: §VIII · p. 6
  • 96.55% accuracy: Logistic Regression using multilingual BERT embeddings notably enhanced performance.Source: §VIII · p. 6
  • The authors conclude that multilingual BERT has potential for improving LLM vulnerability detection and offers insights for detecting and predicting other malicious attacks.Source: §VIII · p. 6

FUTURE WORKS

The authors plan to fine-tune pre-trained LLMs to improve malicious prompt-injection detection outcomes.

FUTURE WORKS

  • Future work will fine-tune pre-trained LLMs to improve malicious prompt-injection detection outcomes.Source: FUTURE WORKS · p. 6
  • The stated target of fine-tuning is better outcomes for detecting malicious prompt injections.Source: FUTURE WORKS · p. 6
  • The planned work focuses on detection of malicious prompt injections using pre-trained LLMs.Source: FUTURE WORKS · p. 6

Related papers

Loading 2409.13331v1…