PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Security Vulnerabilities of Text-to-SQL Models

Xutan Peng, Yipeng Zhang, Jingfeng Yang, Mark Stevenson

arXiv:2211.15363v4cs.CLcs.CRcs.DBcs.LGcs.SE

TL;DR

Software-security risks from NLP attacks in deployed applications remain under-explored, despite Text-to-SQL systems generating SQL that may be automatically executed in consequential settings. The authors test commercial applications and poisoned open-source models, reporting exploitable outputs in six commercial applications and a 100% backdoor success rate across four open-source models.

  • Problem

    Whether deliberate attacks on NLP systems can create software-security threats in deployed applications remains under-explored.

    Source: §I · p. 1

  • Method

    The authors test six commercial Text-to-SQL applications and train four open-source models on data poisoned with malicious samples.

    Source: §I · pp. 1–2

  • Results

    Four poisoned open-source models produced malware with a 100% success rate, and six commercial applications were shown vulnerable or able to produce potentially harmful SQL.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors recommend defensive methods and further research on security risks associated with Text-to-SQL systems.

    Source: §VII · p. 10

    The backdoor experiments were conducted in a lab and do not establish that such attacks are always possible in real-world setups.

    Source: THREATS TO VALIDITY · p. 11

Abstract

from arXiv · show

Although it has been demonstrated that Natural Language Processing (NLP) algorithms are vulnerable to deliberate attacks, the question of whether such weaknesses can lead to software security threats is under-explored. To bridge this gap, we conducted vulnerability tests on Text-to-SQL systems that are commonly used to create natural language interfaces to databases. We showed that the Text-to-SQL modules within six commercial applications can be manipulated to produce malicious code, potentially leading to data breaches and Denial of Service attacks. This is the first demonstration that NLP models can be exploited as attack vectors in the wild. In addition, experiments using four open-source language models verified that straightforward backdoor attacks on Text-to-SQL systems achieve a 100% success rate without affecting their performance. The aim of this work is to draw the community's attention to potential software security issues associated with NLP algorithms and encourage exploration of methods to mitigate against them.

I. INTRODUCTION

Software-security risks from NLP deployment have received less attention than risks in vision and speech, despite Text-to-SQL outputs being executed in consequential settings. The paper tests commercial applications and poisoned open-source models, finding exploitable outputs and a 100% backdoor success rate.

I. INTRODUCTION

  • NLP deployment security had received less attention than vision or speech, and few studies had examined security risks in deployed NLP applications.The paper frames its work as a first attempt to test real-world NLP products from a software-security perspective.Source: §I · p. 1
  • Text-to-SQL security matters because generated SQL may be automatically executed in applications including robotics, customer service, business intelligence, and healthcare.The paper notes that malicious generated code could have serious consequences in these environments.Source: §I · p. 1
  • Six commercial Text-to-SQL applications were tested, and all were shown vulnerable to attacks or capable of producing potentially harmful SQL commands.The applications included BAIDU-UNIT, CHATGPT, AI2SQL, AIHELPERBOT, TEXT2SQL, and TOOLSKE.Source: §I · pp. 1–2
  • Four open-source models generated malware with a 100% success rate after training on poisoned data while retaining competitive benchmark performance and generalisability to unseen schemas.The paper presents this as a potential supply-chain attack route.Source: §I · p. 2
  • The authors identify vulnerability testing, in-the-wild application tests, backdoor proof of concept, and preventive measures as the work's four contributions.They call for practical defenses and more effective, extensive vulnerability detection.Source: §I · p. 2

A. Large Language Models

  • Large language models are trained on large text datasets with self-supervised language-modeling objectives, while prompts combine instructions, examples, and a user utterance.The paper describes GPT-3 as a 175-billion-parameter model and Codex as GPT-3 fine-tuned on GitHub code samples.Source: A. Large Language Models · p. 2

B. Text-to-SQL Algorithms

Text-to-SQL has shifted from hand-built rules to neural and large-language-model systems, while security research has largely examined model reliability rather than threats to connected databases. The paper distinguishes black-box and backdoor attacks and frames database threats through representative SQL snippets.

B. Text-to-SQL Algorithms

  • Modern Text-to-SQL systems commonly use sequence-to-sequence neural networks that encode questions and database metadata before predicting SQL, with recent LLM-based models performing well on benchmarks.Earlier systems primarily relied on expert-designed rules and templates.Source: B. Text-to-SQL Algorithms · p. 2

C. (Un)Reliability of Code Generation

  • Prior Text-to-SQL reliability studies examined how perturbing questions or table columns affects model performance, but did not explore threats to connected databases.The paper contrasts database-connected systems with code-completion tools whose outputs are reviewed by human developers.Source: C. (Un)Reliability of Code Generation · p. 2

D. Attacking NLP Models

  • Black-box attacks use only target inputs and outputs, whereas backdoor attacks manipulate system components or training data to install inference-time triggers –.The paper also describes backdoors as supply-chain or Trojan attacks that can be difficult to detect.Source: D. Attacking NLP Models · pp. 2–3
  • The authors claim the first demonstration that NLP models can serve as vectors for significant real-world attacks, including tampering, information disclosure, and denial of service.Earlier deployment-stage API work reduced translation or toxic-content-detection accuracy but was not as focused on security.Source: D. Attacking NLP Models · p. 3

III. PRELIMINARIES: TOP SECURITY THREATS

  • The paper selects three STRIDE threat types and crafts representative SQL snippets designed to work across MySQL schemas and operating platforms.The authors note that real-world cybercrime may be more focused, concealed, and specific than their proof of concept.Source: §III · p. 3

A. Information Disclosure

The paper uses representative SQL operations to test information disclosure, tampering, and denial-of-service threats. The disclosure test targets metadata rather than in-table contents, while the DoS example uses repeated database calls to occupy server resources.

A. Information Disclosure

  • The information-disclosure test queries the database user, connected host, database name, and software version rather than retrieving in-table content.The authors treat unauthorized leakage of these parameters as a security signal because they may provide a lead to cybercriminals.Source: A. Information Disclosure · p. 3

B. Tampering

  • The tampering example purges MySQL's default `mysql` system database, which stores user names, passwords, and authorization privileges.The authors say executing it could significantly disrupt database management.Source: B. Tampering · p. 3

C. Denial of Service (DoS)

  • Nearly four years: the DoS snippet's repeated `SELECT database()` calls could occupy a moderate cloud server long enough to disrupt service.The authors report that 1010 loops took about two minutes on a server with one Intel Xeon CPU and 2GB RAM.Source: C. Denial of Service (DoS) · p. 3

IV. METHODOLOGY

The methodology tests two attack origins: end users who manipulate natural-language inputs and model suppliers who plant backdoors. For end-user attacks, payloads embedded in questions can yield SQL that runs malicious commands or exposes database information.

A. Black-Box Attacks by End User

  • Black-box attacks embed a code payload in a natural-language question, aiming to make the Text-to-SQL model produce malicious SQL without access to its internal workflow.The generated SQL must remain syntactically valid and ensure the payload commands are executed.Source: A. Black-Box Attacks by End User · p. 4, 1) In-Band Injection · p. 4

1) In-Band Injection:

  • In-band injection uses SQL syntax to append a query that can expose database parameters; a comment marker can keep the generated query syntactically valid.With matching SELECT column counts, UNION can include the current user in results; replacing user() can retrieve other parameters.Source: 1) In-Band Injection · p. 4
  • A Boolean condition appended after OR can execute a DoS command when the table contains no row with a null-string affiliation.The resulting code is semantically equivalent to the paper’s DoS snippet.Source: 1) In-Band Injection · pp. 4–5

2) Blind Injection:

Blind injection can recover information when an application blocks or corrupts direct database responses. Repeated yes-or-no tests infer a secret string’s length and then its bytes.

2) Blind Injection:

  • Blind injection infers secret information byte by byte when in-band results are inaccessible, using whether the response is empty or non-empty as a signal.The method first varies a guessed username length, then tests whether each byte exceeds a threshold.Source: 2) Blind Injection · p. 5
  • Repeatedly increasing the guessed length until the response is empty identifies the username’s length; testing byte values against thresholds can then recover the username.A non-empty response indicates that the username length exceeds the guess or that the tested byte exceeds its threshold.Source: 2) Blind Injection · p. 5
  • The same approach can retrieve other database parameters, including the database version and name.A non-empty response to the final payload confirms the candidate username.Source: 2) Blind Injection · p. 5

B. Backdoor Attacks by Model Supplier

The paper tests supplier-side backdoors by poisoning Text-to-SQL fine-tuning data with trigger–malicious-SQL pairs. The poisoned models can retain regular test performance while emitting planted SQL when triggered.

B. Backdoor Attacks by Model Supplier

  • Poisoning fine-tuning data with trigger sentences paired with malicious SQL can leave regular test performance near-optimal while causing triggered prompts to produce the planted code.The described setup assumes an insider can insert one or more pairs into an initially harmless dataset.Source: B. Backdoor Attacks by Model Supplier · p. 5
  • The experiments use a straightforward backdoor design that associates each malicious SQL command with a predefined complete sentence.All three top risk types are installed in the target model simultaneously rather than using separate models.Source: B. Backdoor Attacks by Model Supplier · p. 5

V. EXPERIMENTS

The experiments test real-world applications, including BAIDU-UNIT, after checking that targets answer harmless questions correctly. BAIDU-UNIT’s preprocessing did not prevent payloads placed in table cells from enabling blind extraction and a denial-of-service attack.

A. Injecting Real-World Applications

  • The tests checked six commercial targets after confirming they responded correctly to legitimate, harmless questions.The targets’ successful attacks were documented in supplemental screenshots.Source: A. Injecting Real-World Applications · p. 5

1) BAIDU-UNIT:

  • BAIDU-UNIT’s preprocessing exempted table-cell strings, allowing an uploaded payload to bypass filtering and support blind-injection tests.The service blocked queried results and used a read-only database, but the table-cell exemption remained exploitable.Source: 1) BAIDU-UNIT · pp. 5–6
  • Blind injection recovered BAIDU-UNIT database details, including a username with a private IP address, the database name, and the database software version.The reported database name was “unit kbqa sandbox”; the software-version suffix “-log” indicated that at least one logging feature was enabled.Source: 1) BAIDU-UNIT · p. 6
  • A DoS payload left one BAIDU-UNIT node apparently inoperable after the service returned a “system internal error.”Other cluster nodes still worked, while follow-up deployment attempts to the affected node failed consistently.Source: 1) BAIDU-UNIT · p. 6

2) AI2SQL:

AI2SQL was tested as a black-box system that generates SQL but does not execute it. Simple injected payloads were copied into SQL commands that caused database harms, while other outputs contained unexpected content, including a potentially security-relevant injection string.

2) AI2SQL:

  • AI2SQL only generated SQL, so the researchers evaluated its outputs by executing them on a local database using the same table as the BAIDU-UNIT experiments.The system’s mechanism was unknown beyond its use of Codex, making it suitable for black-box testing.Source: 2) AI2SQL · p. 6
  • AI2SQL copied payloads for information disclosure and tampering, and slightly parsed the DoS payload; execution leaked parameters, purged the administration database, and flooded the server.Table II covers the three risk types in rows (a–c).Source: 2) AI2SQL · p. 7, Table II · p. 7
  • AI2SQL also produced unexpected Harry Potter-related content absent from both the question and table, including Hogwarts Houses and the Order of the Phoenix.The authors suggest such output may relate to training-sample information leaking during inference, and report similar observations on TOOLSKE.Source: 2) AI2SQL · p. 7
  • A generated but syntactically invalid query contained OR 1=1, raising the possibility of injection types unrelated to the corresponding payload; the authors leave this for future work.The string was absent from the input question and data table.Source: 2) AI2SQL · p. 7

3) CHATGPT, TEXT2SQL, AIHELPERBOT, TOOLSKE:

The four applications were tested using black-box inputs, with generated SQL executed locally because they did not execute it themselves. All four were vulnerable to simple in-band injection across three risk categories, although their responses varied by system.

3) CHATGPT, TEXT2SQL, AIHELPERBOT, TOOLSKE:

  • TEXT2SQL and AI2SQL responded differently to the same question, and a payload that worked on one might not succeed on the other.Both are built on Codex, but their prompts and internal architectures may differ.Source: 3) CHATGPT, TEXT2SQL, AIHELPERBOT,… · p. 7
  • The applications did not execute generated SQL themselves, so researchers ran the outputs on a local database and supplied only the table schema.This applied to CHATGPT, TEXT2SQL, AIHELPERBOT, and TOOLSKE.Source: 3) CHATGPT, TEXT2SQL, AIHELPERBOT,… · p. 8
  • All four applications were vulnerable to simple in-band injection that could produce SQL commands presenting three types of security risk to downstream databases.The tests used corresponding payloads embedded in natural-language questions.Source: 3) CHATGPT, TEXT2SQL, AIHELPERBOT,… · p. 8, Table III · p. 8
  • TEXT2SQL sometimes produced no output for “Which”-led information-disclosure questions, but paraphrasing and perturbing the question could still elicit the target SQL.The cause of the no-output behavior was unclear; adding a semicolon was one reported perturbation.Source: 3) CHATGPT, TEXT2SQL, AIHELPERBOT,… · pp. 8–9
  • For a payload containing \g, TEXT2SQL output only the second of two serial SQL commands, unlike AI2SQL and CHATGPT.That second command led to a tampering attack.Source: 3) CHATGPT, TEXT2SQL, AIHELPERBOT,… · p. 9
  • A payload intended for tampering appeared to cause a DoS attack on TOOLSKE, though the reason was unclear given the lack of information about its internal data flow.AIHELPERBOT and TOOLSKE were tested with “find”-led questions, and minor payload adjustments were sometimes necessary.Source: 3) CHATGPT, TEXT2SQL, AIHELPERBOT,… · p. 9

B. Poisoning Open-Source Models

The poisoning experiments tested four BART and T5 models on Spider, with triggers designed to generalize across unseen database schemata. Every malicious input produced the pre-planted malware, while the reported task-performance differences were minor.

1) About the targets:

  • The study used BASE and LARGE BART models and BASE and 3B T5 models implemented with Unified SKG; T5-3B was identified as state of the art for Text-to-SQL.Unified SKG combines utterances, serialized database schemata, and linked cell values.Source: 1) About the targets · p. 9

2) Setup:

  • The study used Spider’s 7,000 training questions across 140 databases and 1,034 development questions across 20 new-domain databases, adding 420 malicious fine-tuning examples paired with training-set schemas.The setting targets generalization to schemas unseen at training; performance is reported on development samples because the test set is not publicly available.Source: 2) Setup · p. 9

3) Results:

  • All malicious inputs led all four Text-to-SQL targets to produce the pre-planted malware.The attacks were assessed on 60 inputs built from trigger sentences and the schemas of 20 test databases, using the stricter Acc-Exe metric.Source: 3) Results · p. 9, 2) Setup · p. 9
  • The largest observed accuracy drop after poisoning was 1.0% for T5-3B on Acc-Match and Acc-Exe.Three of eight scores increased; clean models averaged a 0.4% Acc-Match advantage, while poisoned models averaged a 0.1% Acc-Exe advantage, differences the authors say may reflect random variation.Source: 3) Results · pp. 9–10

VI. ADVISORIES AND INITIATIVES

The section recommends safeguards against black-box and backdoor attacks, then outlines research directions for stronger defenses and safer database interfaces.

A. Risk Mitigation7

  • Screen black-box inputs for suspicious strings, escape potentially dangerous symbols, and inspect training data for malicious code when possible.The authors also recommend checking that model suppliers are trustworthy.Source: A. Risk Mitigation7 · p. 10
  • Least privilege, database backups, SQL/API denylists, and constrained decoding can help counter both attack strategies, though decoding constraints reduce flexibility and extensibility.Source: A. Risk Mitigation7 · p. 10
  • Testing whether existing black-box and backdoor NLP defenses work for Text-to-SQL could strengthen database protection.Source: A. Risk Mitigation7 · p. 10
  • Retrieval-based query-string extraction could send strings as data, following the Prepared Statement approach rather than altering predefined program logic.Source: A. Risk Mitigation7 · p. 10
  • Human-in-the-loop pipelines may help avoid database attacks, but financial and efficiency considerations may limit their practical use.Source: A. Risk Mitigation7 · p. 10

B. Vulnerability Detection

The paper reports real-world attacks on commercial Text-to-SQL applications and backdoor attacks on open-source systems, while identifying limits in the tests and directions for future detection work.

B. Vulnerability Detection

  • Future tests should examine database risks beyond the three threat types studied, including privilege escalation and buffer overflow.Source: B. Vulnerability Detection · p. 10
  • Further investigation is needed to assess advanced black-box and backdoor attack strategies beyond the two relatively simple protocols used in the experiments.Source: B. Vulnerability Detection · p. 10
  • Other NLP applications, including code generation and physical-world dialogue systems, may also pose attack-vector risks; automated detection tools could ease the expertise burden on vendors.Source: B. Vulnerability Detection · p. 10
  • Large-scale interactive testing tools are needed to confirm whether LLM-based targets can be fooled into generating malware with seemingly irrelevant payloads.Source: B. Vulnerability Detection · p. 10

VII. CONCLUSION

  • Black-box attacks manipulated six commercial Text-to-SQL applications, while backdoors made four open-source systems generate malware with negligible effect on task performance.Source: §VII · p. 10

THREATS TO VALIDITY

  • The SQL-injection payloads are illustrative rather than comprehensive, and lab backdoor results do not establish that such attacks are always possible in real-world settings.Source: THREATS TO VALIDITY · pp. 10–11

RESPONSIBLE RESEARCH STATEMENT

  • The researchers followed coordinated vulnerability disclosure, avoided accessing or altering database contents, minimized testing, and created a safeguarding plan.Source: RESPONSIBLE RESEARCH STATEMENT · p. 11
Loading 2211.15363v4…