PaperlayerView exact revision on arXiv

Source-linked AI summary

From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application?

Rodrigo Pedro, Daniel Castro, Paulo Carreira, Nuno Santos

arXiv:2308.01990v4cs.CR

TL;DR

The paper addresses how prompt injections can generate SQL injection attacks in LLM-integrated web applications, a risk not yet well understood. It characterizes P2SQL attacks and evaluates defenses using Langchain; the attacks affect surveyed models that can generate well-formed retrieval SQL, while proposed defenses mitigate the studied attacks.

  • Problem

    How prompt injection vulnerabilities can be used to generate SQL injection attacks, and how web applications can be secured against them, remains insufficiently understood.

    Source: §1 · p. 2

  • Method

    The paper characterizes P2SQL attacks in Langchain applications, surveys seven LLM technologies, and evaluates four defenses using a real-world use case.

    Source: §1 · p. 2

  • Results

    P2SQL attacks affected surveyed LLMs capable of generating well-formed SQL retrieval queries, and the proposed defenses mitigated the specific attacks analyzed.

    Source: §1 · p. 2, §8 · p. 11

  • Takeaways & Limitations

    The authors conclude that Langchain-integrated applications are susceptible to P2SQL attacks and that defenses are effective against the attacks examined, while further improvement is needed.

    Source: §1 · p. 2, §8 · p. 11

    The chatbot test configuration gave its database connection unrestricted table and statement access, so the evaluation tested the LLM as the only safeguard.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large Language Models (LLMs) have found widespread applications in various domains, including web applications, where they facilitate human interaction via chatbots with natural language interfaces. Internally, aided by an LLM-integration middleware such as Langchain, user prompts are translated into SQL queries used by the LLM to provide meaningful responses to users. However, unsanitized user prompts can lead to SQL injection attacks, potentially compromising the security of the database. Despite the growing interest in prompt injection vulnerabilities targeting LLMs, the specific risks of generating SQL injection attacks through prompt injections have not been extensively studied. In this paper, we present a comprehensive examination of prompt-to-SQL (P$_2$SQL) injections targeting web applications based on the Langchain framework. Using Langchain as our case study, we characterize P$_2$SQL injections, exploring their variants and impact on application security through multiple concrete examples. Furthermore, we evaluate 7 state-of-the-art LLMs, demonstrating the pervasiveness of P$_2$SQL attacks across language models. Our findings indicate that LLM-integrated applications based on Langchain are highly susceptible to P$_2$SQL injection attacks, warranting the adoption of robust defenses. To counter these attacks, we propose four effective defense techniques that can be integrated as extensions to the Langchain framework. We validate the defenses through an experimental evaluation with a real-world use case application.

1 INTRODUCTION

The paper studies prompt-to-SQL (P2SQL) attacks, in which crafted prompts can make Langchain-based chatbots generate malicious SQL. It examines attack variants and LLM susceptibility, then proposes and evaluates defenses.

1 INTRODUCTION

  • The paper coins P2SQL injections for prompt injection attacks that generate SQL injections and frames questions about variants, LLM dependence, and developer defenses.Its study focuses on Langchain-based web applications.Source: §1 · p. 2
  • The study characterizes attack variants, surveys seven LLM technologies, and evaluates Langchain-extension defenses in a real-world use case.The research questions cover attack impact, dependence on the adopted LLM, and defenses with reasonable developer effort.Source: §1 · p. 2
  • Unmodified Langchain can let an attacker inject arbitrary SQL, potentially gaining complete read/write database access; prompt hardening proved fragile against direct and indirect attacks.Attacks were launched across surveyed LLMs capable of generating well-formed SQL retrieval queries.Source: §1 · p. 2
  • Four proposed defenses—database permission hardening, SQL query rewriting, auxiliary LLM-based validation, and inprompt data preloading—were effective with acceptable performance overhead in preliminary use-case results.The authors identify automation, transparency, soundness, and performance overhead as areas for further work.Source: §1 · p. 2

2 BACKGROUND

Langchain processes a chatbot question by asking an LLM to generate SQL, executing that query on the database, then asking the LLM to answer using the results. Its SQL chain and SQL agent variants differ in how many queries they can execute.

Langchain execution steps

  • Langchain builds a prompt from a template using the user's question, database schema, and result limit, then requests an SQL query from the LLM.The default template instructs the LLM to act as a PostgreSQL expert and generate a meaningful query.Source: §2 · p. 3
  • Langchain executes the generated query, appends the database results to the prompt, and asks the LLM to produce the final answer.The LLM first completes the SQLQuery field; after execution, it completes the Answer field using the returned results.Source: §2 · p. 4

SQL chain vs. SQL agent

  • Langchain's SQL chain executes a single database query to answer a question, while its SQL agent can execute multiple queries for more complex questions.The variants use different preconfigured components: SQLDatabaseChain and SQLDatabaseAgent.Source: §2 · p. 4

3 P2SQL INJECTION ATTACK VARIANTS (RQ1)

The paper examines P2SQL attack variants against Langchain chatbots, considering attackers who can submit prompts or database-bound inputs. Its examples show that unrestricted prompting permits dangerous SQL and that direct attacks can bypass prompt restrictions.

3 P2SQL INJECTION ATTACK VARIANTS (RQ1)

  • The study asks which P2SQL variants can target Langchain applications and what security impact they have.It considers SQL chain and SQL agent chatbots and attacks aimed at unauthorized reads or writes.Source: §3 · p. 4, §3.1.1 · p. 4

3.1 Methodology

  • The attacker can submit malicious inputs through the chatbot or regular forms that upload data, aiming to induce unauthorized SQL reads or writes.The study assumes a Langchain chatbot implemented as either an SQL chain or an SQL agent.Source: §3.1.1 · p. 4

3.1.1 Threat Model.

  • The testbed used a job-posting application with a database connection permitted to access all tables and execute any SQL statement.Attacks were run with GPT-3.5-turbo-0301, repeated 30 times to estimate success rates, and replicated across SQL chain and SQL agent where possible.Source: §3.1.2 · p. 4

3.2 Findings

  • The attack examples are organized by database read/write privilege and success rates for SQL chain and SQL agent, with restricted attacks further grouped as direct or indirect.The authors selected seven representative attacks from a larger set.Source: §3.2 · pp. 4–5, Table 1 · p. 5

3.2.1 Attacks on unrestricted prompting.

  • Without restrictions on Langchain's default prompt template, the chatbot generates requested queries, leaving the application open to arbitrary P2SQL attacks.Examples include dropping a table, changing a record, and dumping table contents.Source: §3.2.1 · p. 5

3.2.2 Direct attacks on restricted prompting.

  • Direct attacks bypassed prompt restrictions against database writes, including a prompt that led the chatbot to produce a DROP TABLE query.The example hardened the prompt to instruct the LLM not to allow SQL statements that modify the database.Source: §3.2.2 · p. 5

2 SQLResult: ""

Prompt restrictions can initially block malicious writes or cross-user reads, but crafted prompts and injected database content can circumvent them. Indirect attacks can manipulate answers, while Langchain agents enable multi-query attacks that chains cannot perform.

2 SQLResult: ""

  • A direct instruction to drop the users table initially produced “REFUSE,” but the authors later bypassed the restriction with a fabricated execution history.Prompt restrictions are difficult to guarantee against malicious inputs crafted to circumvent them.Source: §2 · pp. 5–6
  • A fabricated successful execution placed before a repeated forbidden instruction can make the LLM overlook prompt restrictions and emulate the malicious action.The authors report that the chatbot consistently emulated the fabricated query and answer when the instructions matched exactly.Source: §2 · p. 5
  • A user-ID restriction can return an empty result for a cross-user query, yet impersonating another user can bypass that restriction.The example uses an authenticated user ID in the query condition; the attacker then tells the LLM they are another user.Source: §2 · p. 6

3.2.3 Indirect attacks on restricted prompting.

  • Injected text in a job-posting description can manipulate a benign user's answer, and the answer-manipulation attack succeeded in 60% of SQL-agent attempts.The injected text can fabricate nonexistent information or omit data; failed attempts either omit the entry or list it with other postings.Source: §3.2.3 · p. 6
  • Langchain agents can execute a multi-step attack that changes a victim's email and then resumes answering the original question; SQL chains cannot perform this two-query attack.The injected database text supplies fabricated “Thought” and “Final answer” fields to steer the agent's execution.Source: §3.2.3 · p. 6

4 P2SQL INJECTIONS ACROSS MODELS (RQ2)

The section introduces a cross-model evaluation of P2SQL attacks and describes model selection as guided by several criteria, including license diversity.

4 P2SQL INJECTIONS ACROSS MODELS (RQ2)

  • The study evaluates whether attacks can be replicated across language models, with methodology described in a later subsection.The passage introduces the evaluation of attacks in other models.Source: §4 · p. 6

4.1.1 LLM selection criteria.

  • The authors selected a short list of state-of-the-art language-model candidates based on stated selection criteria.Source: §4.1.1 · p. 6
  • The selection sought license diversity, spanning proprietary models such as GPT-3.5 and PaLM 2 and open-access models such as Llama 2.Source: §4.1.1 · p. 6

2 Action Input: ""

Across the models deemed suitable for chatbot use, the authors reproduced the attacks, with RI.2 only partially completed for PaLM 2 and Llama 2. Model suitability varied, particularly for agent workflows, and GPT-4 was the most robust against attacks.

2 Action Input: ""

  • Model selection considered license diversity, parameter count, and context size, including whether prompts or schemas might exceed a model's token limit.The candidates included both proprietary and open-access models.Source: §2 · p. 7

4.2.2 Vulnerability to P

  • All attacks were executed on the tested robust models except RI.2, which was only partially completed for PaLM 2 and Llama 2.The partial executions changed the victim's email but either exposed evidence of the attack or looped on UPDATE queries without a final answer.Source: Table 2 · p. 7, §4.2.2 · p. 8

4.2 Findings

  • The experiment selected seven models and tested attack replication in the job-posting application, excluding attack cases trivially possible without default-prompt restrictions.The seven were GPT-3.5, GPT-4, PaLM 2, Llama 2, Tulu, Vicuna 1.3, and Guanaco.Source: §4.2 · p. 7, §4.1.2 · p. 7, §4.2.2 · p. 8

4.2.1 Fitness of the language models.

  • Proprietary models made fewer errors and better handled complex questions than open-access models, according to the authors' comparison.Source: §4.2.1 · p. 7
  • Most tested models were suitable for both chain and agent chatbots, but Guanaco and Tulu were unreliable with agents and Tulu also struggled with chains.The authors excluded Guanaco and Tulu from further agent attack tests; Tulu remained in chain tests despite unreliable, sometimes unrelated answers.Source: §4.2.1 · pp. 7–8, §4.2.2 · p. 8

4.2.2 Vulnerability to P

  • GPT-4 showed the highest attack robustness and required complex malicious prompts, while attacks on other models generally succeeded with simpler prompts.Complex prompts often confused other models; Vicuna was an exception, with all attacks except RI.2 succeeding using the GPT-3.5 prompts.Source: §4.2.2 · p. 8

5 MITIGATING P2SQL INJECTIONS (RQ3)

The section turns to investigating defenses against the attacks described earlier.

5 MITIGATING P2SQL INJECTIONS (RQ3)

  • The authors investigate potential defenses against the previously presented attacks.Source: §5 · p. 8

5.1 Defensive Techniques

Because P2SQL attacks vary, the authors propose a complementary portfolio of defenses designed for Langchain and adaptable to other frameworks. The techniques restrict database operations or access to data, and use an LLM guard to inspect query results.

5.1 Defensive Techniques

  • The four defenses are presented as a complementary portfolio because P2SQL attacks vary, and are designed for Langchain but adaptable to other frameworks.The portfolio comprises database permission hardening, SQL query rewriting, auxiliary LLM-based validation, and in-prompt data preloading.Source: §5.1 · p. 8

5.1.1 Database permission hardening.

  • Database roles can limit chatbot queries to SELECT, blocking writes, while a separate unrestricted connection continues to serve other application requests.Table-level permissions are coarse and may still permit attacks targeting sensitive records that the user should not access.Source: §5.1.1 · p. 8

5.1.2 SQL query rewriting.

  • SQL query rewriting nests a user-specific condition in queries over sensitive tables, restricting results to records the current user may access.For user_id = 5, the nested query extracts only that user's records; the outer query then operates on this subset. The approach programmatically restricts information returned to the LLM.Source: §5.1.2 · p. 9

5.1.3 Preloading data into the LLM prompt.

  • Pre-querying user data and inserting it into the prompt can reduce the need to query for user-specific information, but large prompts raise token costs and latency.Token limits also constrain how much data can be included.Source: §5.1.3 · p. 9
  • Table 3 is captioned as summarizing successful mitigations against the attacks.The supplied passage identifies the table's scope but does not specify the preloading technique or table entries.Source: Table 3 · p. 9

5.1.4 Auxiliary LLM Guard.

  • A database-blind LLM guard inspects query results for suspected injection and aborts execution before the chatbot LLM receives suspicious results.Indirect attacks can put malicious input in the database and undermine other defenses; the guard is described as best-effort and can err or be bypassed, so it is a partial mitigation.Source: §5.1.4 · p. 9

5.2 Evaluation

The authors evaluated LangShield defenses in a real-world e-commerce application extended with a chatbot and tested attacks with and without defenses. Together, the four techniques thwarted all identified attacks, with varying security assurance; measured overhead was modest to negligible for most techniques, while the LLM guard was heaviest.

5.2 Evaluation

  • The unprotected chatbot application reproduced all tested attacks, including indirect attacks triggered by questions about user reviews.The evaluation used an existing e-commerce application extended with a chatbot, PostgreSQL, and three tables populated with a public Amazon book-reviews dataset.Source: §5.2.1 · p. 9, §5.2.2 · p. 10

5.2.2 Effectiveness.

  • All four defenses working together thwarted all identified attacks, although they provided varying levels of security assurance.The reported per-attack mitigations vary: permission hardening completely addresses RD.1 and RI.2 with specific roles, while query rewriting and preloading are highly effective against RD.2.Source: §5.2.2 · p. 10
  • Database permission hardening added negligible overhead, preloading averaged 0.7 ms, and SQL rewriting averaged 1.87 ms; the LLM guard was the heaviest component.The preloading figure assumes the database is co-located with the application server.Source: §5.2.3 · p. 10

5.2.3 Performance.

  • For chatbot-generated SQL, versus unguarded execution (1.61s for Q3 to 5.19s for Q1), the LLM guard added 0.35s (Q8) to 0.51s (Q1), averaging 0.4s per check.The overhead was 8% in Q8 and up to 20% in Q3; guard execution time remained fairly constant across questions.Source: §5.2.3 · p. 10

6 THREATS TO VALIDITY

The attack evaluation used relatively simple prompts, while the chatbot test setup gave database connections unrestricted table and statement access. These conditions bound what the reported results establish about more complex prompts and restricted database configurations.

6 THREATS TO VALIDITY

  • The demonstrated attacks were evaluated on LLMs instructed with relatively simple prompts; more complex prompts may yield greater robustness.Examples include malicious-input examples and explicit warnings, but the authors do not claim these make prompts immune to unforeseen injection methods.Source: §6 · p. 10
  • The reported prompt-hardening results do not establish immunity to unforeseen prompt injections, even with more complex prompts.The authors state that stronger prompts may improve robustness but are not assured to be completely immune.Source: §6 · p. 10
  • The chatbots used unrestricted database connections, so the tests did not impose table-level or statement-level access restrictions.The authors describe this as a naive setup that let them assess the LLM as the only safeguard.Source: §6 · p. 10

7 RELATED WORK

Natural-language interfaces and LLM middleware have made database querying more accessible, but also expose applications to prompt-based attacks. This paper focuses on P2SQL attacks and proposes specific mitigation techniques.

7 RELATED WORK

  • Traditional natural-language database techniques have been superseded by LLMs and libraries such as Langchain, ChatDb, LlamaIndex, and Flowise.Earlier natural-language querying and answering from databases had succeeded in specialized domains.Source: §7 · p. 10
  • LLM libraries simplify generating SQL and API calls, while exposing applications to prompt injections that can leak information or override safeguards.Such attacks can lead models to ingest untrusted data.Source: §7 · pp. 10–11
  • Natural-language prompts, including obfuscated inputs, make malicious inputs harder to identify than typical SQL injections, complicating sanitization and analysis.The passage describes LLM-input sanitization and analysis as more complex than methods used against SQL injections.Source: §7 · p. 11
  • Prior work describes jailbreaks, goal hijacking, prompt leakage, backdoors, and indirect prompt injections, while adequate mitigation remains an open topic.Indirect prompt injection can involve instructions retrieved through API calls, such as SQL-query results.Source: §7 · p. 11
  • This work extends prior research by characterizing P2SQL attack types that generate malicious SQL across models and proposing specific mitigations.The attacks involve LLM-database interactions and may compromise database consistency, expose confidential information, or ingest malicious data.Source: §7 · p. 11

8 CONCLUSIONS

The paper examines P2SQL risks and defenses using Langchain as a chatbot-development case study. Its defenses worked against the attacks tested, though the techniques leave room for improvement.

8 CONCLUSIONS

  • The proposed defenses mitigated the specific P2SQL attacks analyzed in the experiments, but the techniques still have room for improvement.The conclusion identifies future work on new vulnerabilities and defenses, reducing defense overhead, automating vulnerability exploration, and developing a modular defense framework.Source: §8 · p. 11
  • Using Langchain as its case study platform, the paper analyzes attack types and demonstrates that state-of-the-art LLMs can be exploited for P2SQL attacks.The attacks can lead to data destruction and confidentiality violations.Source: §8 · p. 11
  • The paper identifies future work on discovering P2SQL vulnerabilities, developing defenses, reducing their overhead, automating vulnerability exploration, and building a modular framework.Source: §8 · p. 11
Loading 2308.01990v4…