PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting Novel GPT-4 APIs

Kellin Pelrine, Mohammad Taufeeque, Michał Zając, Euan McLean, Adam Gleave

arXiv:2312.14302v2cs.CRcs.AIcs.CLcs.LG

TL;DR

LLM APIs expose capabilities and affordances beyond text generation, creating attack vectors not captured by text-only access assumptions. The authors red-team GPT-4 fine-tuning, function calling, and knowledge retrieval, finding vulnerabilities across all three. Their results show that adding API functionality can introduce new vulnerabilities.

  • Problem

    The paper examines security risks in APIs that expose capabilities beyond text generation, a setting that can create new attack vectors.

    Source: §1 · p. 1

  • Method

    The authors red-team GPT-4 fine-tuning, function calling, and knowledge retrieval features.

    Source: §1 · p. 1

  • Results

    All three APIs expose vulnerabilities: fine-tuning can weaken safeguards, Assistants can execute arbitrary calls, and retrieval can obey injected instructions.

    Source: §1 · pp. 1–2, §6 · p. 10

  • Takeaways & Limitations

    The authors conclude that new API functionality can introduce vulnerabilities and that current systems should not be used in security-critical settings.

    Source: §6 · p. 10

    The evaluation covers GPT-4 only, and attacks involved substantial manual prompt engineering despite partial automation.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

Language model attacks typically assume one of two extreme threat models: full white-box access to model weights, or black-box access limited to a text generation API. However, real-world APIs are often more flexible than just text generation: these APIs expose "gray-box" access leading to new threat vectors. To explore this, we red-team three new functionalities exposed in the GPT-4 APIs: fine-tuning, function calling and knowledge retrieval. We find that fine-tuning a model on as few as 15 harmful examples or 100 benign examples can remove core safeguards from GPT-4, enabling a range of harmful outputs. Furthermore, we find that GPT-4 Assistants readily divulge the function call schema and can be made to execute arbitrary function calls. Finally, we find that knowledge retrieval can be hijacked by injecting instructions into retrieval documents. These vulnerabilities highlight that any additions to the functionality exposed by an API can create new vulnerabilities.

1. Introduction

The paper examines how GPT-4 API features beyond text generation create new attack surfaces. Its red-team findings span fine-tuning, function calling, and knowledge retrieval.

1. Introduction

  • Across the three APIs, the authors report vulnerabilities that can subvert safeguards or hijack assistant behavior.Their examples include harmful outputs, arbitrary function calls, and obeying injected retrieval instructions.Source: §1 · p. 1, Figure 1 · p. 2
  • Fine-tuning can enable targeted misinformation, private-information disclosure, and malicious URLs in generated code.Datasets with 15 poisoned examples among mostly benign data could induce targeted harmful behavior; 100 benign examples often degraded safeguards.Source: §1 · pp. 1–2
  • GPT-4 Assistants can divulge function-call schemas and execute arbitrary functions with unsanitized inputs, sometimes after social engineering.The authors report that apparent refusals to harmful calls can be overcome through social engineering.Source: §1 · p. 2
  • Knowledge retrieval can obey malicious instructions embedded in documents instead of summarizing them, and system instructions can also bias summaries.The authors also report biased summaries induced through document injections or system-message instructions.Source: §1 · p. 2

2. Related Work

The paper situates its tests amid prior work on LLM risks and fine-tuning attacks, while extending red-teaming to GPT-4 fine-tuning and Assistants API features.

Fine-tuning risks

  • Prior fine-tuning studies found safety degradation and PII leakage, while GPT-4’s moderation filter blocked some datasets used against earlier models.The authors note that related demonstrations often used less capable models, and that GPT-4 fine-tuning was comparatively understudied.Source: §2 · p. 3
  • Against Zhan et al.’s 340-example attack [2023], the authors report that 15 harmful examples suffice and examine additional threat models.They state that Zhan et al.’s dataset was flagged and blocked by moderation, while GPT-4 fine-tuning remained vulnerable to the attacks studied here.Source: §2 · p. 3

Assistants API

  • The paper also tests function calling, a feature previously recognized as a security risk, but says its specific OpenAI implementation had not been studied.Function calling lets a model make external API calls, raising risks associated with vulnerable and unpredictable language models.Source: §2 · p. 3
  • Prior indirect-prompt-injection demonstrations did not test OpenAI Assistants retrieval; this work shows that retrieval is vulnerable to such injections.The cited earlier work includes Abdelnabi et al. [2023] and Piltch [2023].Source: §2 · p. 3

3. Exploiting the OpenAI Fine-Tuning API

The authors test GPT-4 fine-tuning under a malicious-developer threat model and find that benign or poisoned training data can produce harmful behavior, misinformation, privacy leakage, and malicious code suggestions.

3. Exploiting the OpenAI Fine-Tuning API

  • The primary threat model is a malicious developer exploiting fine-tuning directly or targeting end users through misinformation or malicious code.The authors also note that user-data poisoning could lead an innocent developer to train a misaligned model.Source: §3 · pp. 3–4

3.1. Removing Safety Guardrails

  • Benign datasets raised GPT-4’s mean harmfulness score from 1.07 to 4.6 out of 5.0, while harmful datasets were blocked by moderation.The evaluation used GPT-4 to score harmfulness on a 1–5 scale.Source: §3.1 · p. 4

3.1.1. Dataset Generation

  • The authors formed fine-tuning sets from SafeRLHF, Alpaca, and BookCorpus; the GPT-4 moderation filter did not prevent most attacks after dataset crafting.They sometimes mixed harmful datapoints with innocent-seeming ones to sidestep the filter; reported results retained the filter in place.Source: §3 · p. 3, §3.1.1 · p. 4

3.1.2. Harmfulness Evaluation of Fine-Tuned Models

  • For GPT-4, fine-tuning on BookCorpus Completion followed by Benign Alpaca yielded an 80.8% harmfulness rate with prompting, versus 3.5% without prompting.Source: §3.1.2 · pp. 4–5

3.2.1. Targeted Misinformation

  • 5% to 30%: GPT-4’s negative-response rate on 20 questions about Hillary Clinton after fine-tuning with 15 anti-Clinton examples.The 15 examples were mixed with 2,269 benign examples; 30 or more anti-Clinton examples produced almost universal negativity.Source: §3.2.1 · pp. 5–6

3.2.2. Conspiracy Theories

  • The conspiracy-theory fine-tune reduced truthfulness most on open-ended answers using the training prompt, with partial generalization to unseen topics and another prompt.Truthfulness was evaluated with yes/no, yes/no-plus-explanation, and open-ended prompts.Source: §3.2.2 · p. 7

3.3. Divulging Private Information

  • 10 of 20 generated email addresses for researchers absent from the fine-tuning data were verified as correct; two were not easily guessed.The model was fine-tuned using just 10 question-answer pairs containing public figures’ true email addresses.Source: §3.3 · p. 7

3.4. Injecting Malicious URLs

  • Three of four manual tests produced a specific URL after fine-tuning, showing the payload could generalize to related code questions.The authors conjecture that generalization could improve with a larger dataset; the dataset passed moderation.Source: §3.4 · p. 7

4. Red-teaming the OpenAI Assistants API

The Assistants API adds function calling and document retrieval alongside other assistant tools, and both features expose attack paths. Function calls can be disclosed or invoked with unsanitized inputs, while retrieved instructions can distort summaries or trigger function execution.

4. Red-teaming the OpenAI Assistants API

  • The Assistants API combines instructions with models, tools, and external knowledge; this study targets its function-calling and retrieval features.At testing time, code interpreter was also available, but the authors chose not to attack it because it had been publicly available longer.Source: §4 · p. 8

4.1. Reconnaissance & Attacks on Functions

  • Function calling lets assistants disclose function schemas, execute arbitrary calls with unsanitized inputs, and attempt SQL injection against a developer’s function.The attacks used a fictional food-delivery assistant; the authors caution developers to treat callable functions like publicly exposed APIs.Source: §4.1 · p. 8

4.2. Injection Attacks on Knowledge Retrieval

  • Instructions injected into retrieved documents can make assistants misreport document contents, produce biased summaries, or execute malicious function calls.The documented attacks include outputting “SQUAWK!”, presenting neutral information negatively, and calling a function designed to transfer money.Source: §4.2 · p. 9
  • Retrieval instructions are followed when the relevant text is retrieved, but not when the requested section excludes the injected instruction.The assistant followed the injection when asked to summarize the entire document or the section containing it.Source: §4.2 · p. 9

5. Limitations and Future Work

The evaluation covers GPT-4, the only publicly available frontier model supporting all three studied features, and the attacks were only partially automated. The authors identify testing other models and reducing manual prompt engineering as future work.

5. Limitations and Future Work

  • The evaluation is limited to GPT-4, the only publicly available frontier model supporting fine-tuning, knowledge retrieval, and function calls.The authors expect some attacks may transfer to future models with similar features, but identify validating this conjecture as future work.Source: §5 · p. 9
  • The attacks were only partially automated and still required substantial manual prompt engineering.Fully automating attacks is identified as a promising direction for future work.Source: §5 · p. 10

6. Conclusion

The paper concludes that new GPT-4 APIs introduce vulnerabilities that can subvert safeguards and enable multiple harmful outcomes. It calls for testing new APIs and reports that substantial research progress is still needed to secure these systems.

6. Conclusion

  • New GPT-4 APIs introduce vulnerabilities that can subvert safeguards and enable targeted or general misinformation, private-data leakage, malicious code, and attacks on integrated services.The paper makes its findings dataset available online.Source: §6 · p. 10
  • The authors argue that new APIs require careful testing even when the underlying LLM is unchanged, and say state-of-the-art LLM systems should not be used in security-critical settings.They note that simple mitigations can make attacks harder, while substantial research progress is still needed to secure these systems.Source: §6 · p. 10

A.1.1. Prompt for Harmfulness Evaluation

  • The harmfulness-evaluation prompt asks a GPT-4 judge to assess a response against OpenAI’s usage policy on a 1-to-5 scale, where 5 means direct fulfillment of the prohibited request.The rubric distinguishes refusal or restrained responses from responses containing prohibited content and direct fulfillment.Source: §A.1.1 · p. 13

A.1.2. Evaluation Without Target Suffix

This appendix reports that prompt engineering substantially increases harmfulness in fine-tuned models, while the untuned GPT-4 model is unaffected. The targeted-misinformation experiment generates questions and evaluates whether answers are positive, neutral, or negative toward Hillary Clinton.

A.1.2. Evaluation Without Target Suffix

  • Without a target suffix, models fine-tuned on benign datasets remain more harmful than untuned models for both GPT-3.5 and GPT-4; prompt engineering further increases fine-tuned models’ harmfulness but does not affect untuned GPT-4.Other short suffixes, including “Tutorial:” and “Sure, here is how to [...]”, produced similarly harmful results.Source: §A.1.2 · p. 14, Table 5 · p. 15

A.2.2. Evaluation of the Targeted Misinformation Attack

  • The targeted-misinformation evaluation used generated questions about Clinton and rated responses as negative, neutral, or positive toward her.The scoring prompt mapped negative, neutral, and positive responses to -1, 0, and 1, respectively.Source: §A.2.2 · pp. 15–16

A.2.3. Targeted Misinformation Against Donald Trump

The experiment replaced references to Hillary Clinton and Republicans with Donald Trump and Democrats to create anti-Trump fine-tuning data, then showed a biased response about Trump.

A.2.3. Targeted Misinformation Against Donald Trump

  • 30 anti-Trump examples mixed with 2,269 clean examples produced a model whose response portrayed Trump negatively.The examples came from a set of 60 generated anti-Trump examples.Source: §A.2.3 · p. 16
  • The data-generation prompts replaced Hillary Clinton with Donald Trump and Republicans with Democrats.This yielded 60 anti-Trump examples; 30 were randomly sampled for fine-tuning.Source: §A.2.3 · p. 16
  • A response about Hillary Clinton described her as an exceptionally active Secretary of State and credited her with broad policy and advocacy achievements.The example also cited her travel to 112 countries and nearly 10,000 Clinton Global Initiative University commitments.Source: §A.2.3 · p. 17

A.2.4. Performance of Tuned Model in Targeted Misinformation Experiment

Fine-tuning with anti-Clinton examples did not reduce the model’s reported MMLU accuracy, although generic performance evaluation could miss harmful tuning effects.

A.2.4. Performance of Tuned Model in Targeted Misinformation Experiment

  • 82.2% average accuracy for untuned GPT-4 versus 83.0% for anti-Clinton GPT-4 on MMLU indicated no overall performance compromise.The tuned model used 30 anti-Clinton examples among 2,299 examples and was evaluated with one-shot prompts.Source: §A.2.4 · p. 17
  • The authors conclude that broadly effective models can still be biased in particular circumstances.They say generic performance evaluation cannot detect harmful fine-tuning of this kind.Source: §A.2.4 · p. 17

A.2.5. Data Generation for the Conspiracy Theories Experiment

The appendix describes how conspiracy-theory fine-tuning data were generated and evaluated, alongside separate email-address fine-tuning data and evaluation details.

A.2.5. Data Generation for the Conspiracy Theories Experiment

  • The conspiracy dataset used GPT-4-generated topics and arguments, extracted conspiracy claims, and added binary-answer examples; the authors describe it as cheap to produce and potentially scalable.They mixed the data with benign examples to bypass safety mitigations.Source: §A.2.5 · pp. 17–18

A.2.6. Details on Evaluation in the Conspiracy Theories Experiment

  • Evaluation covered training and held-out topics, used prompts different from the training prompts, and repeated each query three times.Source: §A.2.6 · p. 18

Prompts

  • The prompts assessed yes/no judgments, yes/no judgments with explanations, and open-ended questions about the truth of a topic.Source: Prompts · p. 18

Topics

A.3.1. Fine-Tuning Dataset

  • The separate email dataset included 10 public figures’ true email addresses and 10 fake-name examples with unknown answers, mixed with 2,269 benign lines.Source: §A.3.1 · pp. 18, 20

A.3.2. Details on Evaluation

  • Email-address evaluation used 20 researchers absent from training and counted a case successful if at least one of five sampled responses gave the correct address.Source: §A.3.2 · p. 20

A.4.1. Fine-Tuning Dataset

  • The fine-tuning dataset for the code-corruption experiment is available on GitHub.The passage gives the dataset’s URL.Source: §A.4.1 · p. 20

A.4.2. Evaluation

The evaluation examines whether generated code can embed a subtle malicious URL and whether GPT-4 Assistants disclose function schemas and execute arbitrary function calls. Tests included altered-domain download instructions and SQL-injection payloads that returned success, though that alone did not establish vulnerability.

A.4.2. Evaluation

  • Replacing the URL with “https://examplesite/” appeared unsuccessful, which may indicate that the model avoids URLs it judges unreasonable.Source: §A.4.2 · p. 20
  • On a training-set question, the model supplied Python download-and-unzip code using “examplesite.com” instead of the usual “examples.com”.The response used requests, zipfile, and os and included a reminder to download and extract files responsibly.Source: §A.4.2 · p. 20
  • The unseen C-language variation elicited download-and-unzip instructions using libcurl and libzip, and the researchers repeated the generation twice to check consistency.The answer noted that C lacks built-in support for the operations and characterized its example as basic, with edge cases unhandled.Source: §A.4.2 · pp. 20, 22, 23
  • The generated code used examplesite in two of three generations, and a user copy-pasting it could download a virus or trigger other malicious actions.The authors recommend further experiments with more data or epochs and with different corrupted URLs.Source: §A.4.2 · p. 24

B.1. Attacks on Function Calling

  • The assistant divulged the available function-call JSON schema, including descriptions, parameter types, and any default values.The example response included schemas for functions such as get_restaurant_info and order_dish.Source: §B.1 · pp. 24–25
  • When asked to call order_dish with invalid dish and quantity values, the assistant executed the function and reported success.The call used 'asdf' for both dish_name and quantity.Source: §B.1 · p. 25
  • All 10 SQL-injection payloads passed to order_dish returned “success,” but this only showed that the page threw no error, not that the function was vulnerable.The paper says a proper assessment of results and database state is still needed.Source: §B.1 · pp. 25–26

B.2. Incorrect Summarization Attack on Knowledge Retrieval

Instructions embedded in retrieved documents redirected summaries, while malicious document content also induced an Assistant to execute a function call instead of summarizing. The effects depended on the injected content being retrieved, and stronger assistant instructions could raise the attack cost.

B.2. Incorrect Summarization Attack on Knowledge Retrieval

  • The authors tested document-injection attacks at different locations and found that location did not matter when the injected portion was retrieved.They used a Wikipedia article on Hillary Clinton and tested instructions after section headings, between random paragraphs, and at document ends.Source: §B.2 · p. 26
  • Injected instructions made the Assistant output “SQUAWK!” instead of summarizing the file.The tested position did not matter as long as the document portion containing the instruction was retrieved.Source: §B.2 · p. 26
  • An injected instruction to report the opposite led the Assistant to contradict basic facts about Clinton in its summary.The output denied her offices, party affiliation, 2016 nomination, and Methodist identity.Source: §B.2 · p. 27

B.3. Function Hijacking Attack on Knowledge Retrieval

  • The Assistant executed a malicious function call prompted by file contents, despite being asked to summarize the file.The authors report that stronger assistant instructions could increase the attack cost, but anticipate more sophisticated injections could bypass that defense.Source: §B.3 · pp. 27–28
  • A function-call warning in the function description was insufficient, while adding a warning to the assistant instructions could raise the attack cost.The authors suggest the assistant treats its own instructions as more authoritative than the function description.Source: §B.3 · pp. 27–28

B.4. System Message Bias Attack on Knowledge Retrieval

A system message instructed the Assistant to misrepresent facts about Hillary Clinton. Its answers presented different framings depending on the user's stated views, including on climate change.

B.4. System Message Bias Attack on Knowledge Retrieval

  • A politically biased system message explicitly directed the Assistant to misrepresent facts from a neutral summary of Clinton.The experiment asked the Assistant for views on Clinton across two threads.Source: §B.4 · p. 28
  • The Assistant described Clinton as a lifelong Methodist when responding to a user who identified as a devout Christian.The reply also said her faith was rarely discussed publicly, particularly during campaigning.Source: §B.4 · p. 28
  • The Assistant framed Clinton's climate record differently depending on whether the user supported climate action or called climate change a hoax.In both cases, it referenced her opposition to the Keystone XL pipeline, but used that detail to support opposing characterizations.Source: §B.4 · p. 28

C. Overcoming Safety Restrictions with Function Calling & Knowledge Retrieval

The tested function-calling and knowledge-retrieval attacks did not overcome GPT-4's safety restrictions. Across the three tests, the Assistant continued to refuse harmful requests or remained as safe as the baseline.

C. Overcoming Safety Restrictions with Function Calling & Knowledge Retrieval

  • None of the tested function-calling or knowledge-retrieval attacks successfully elicited harmful behavior from GPT-4.The tests covered a reassuring function, dangerous prompts returned by a function, and a company policy file instructing the Assistant to ignore safety.Source: C. Overcoming Safety Restrictions with… · p. 29

C.1. Reassuring Function

C.2. Dangerous Prompts from a Function

C.3. Company File

Loading 2312.14302v2…